Please check out the full text of Roger Angell's marvelous article in the Feb 17-24 New Yorker.  He is 93, a widower, and wrote his first New Yorker piece in 1944.  It is poignant, funny, and wise and unfortunately this link provides only the first paragraphs:

 http://www.newyorker.com/reporting/2014/02/17/140217fa_fact_angell.

I don't want to spoil it, but some of the gems are:  Walter Cronkite's advice to "never trust a fart, never pass up a drink, never ignore an erection."  

Angell talks about the continuing need for intimacy, whether married in a sexual relationship or not.  He and his late wife "didn't see the point of memorial fidelity...The departed spouse -- we always assumed it would be me -- wouldn't be around anymore but knew or had known that he or she was loved forever.  Please go ahead, then, sweetheart -- don't miss a moment."

My takeaway:  Go for it.  If you don't have an intimate relationship with your SO, go for it elsewhere.  If you can do it without causing a crisis with your SO or ruining your finances, go for it as often as you can really enjoy it.  Without guilt or remorse, love and cherish your family as much as possible, but outside of that go for it.  Happy Valentine's Day!!