I am super attracted to an older man who exudes refinement, sensitivity, and an ability to relax and enjoy life's blessings.  If a lady on your arm is presents herself in a classy manner (nails clean, hair done, modest yet form fitting dress, good jewelry never hurts) then the glances you get will be looks of jealousy - not skepticism.  It isn't that rare of a combination to have an older man and a younger woman.  

However, if the lady looks disheveled and can't take care of herself - then it's just an eyesore and people will notice.  Choose carefully!  There are plenty of great women out there.

Also, another note is that a previous gentleman once told me that he was uncomfortable with the amount of looks simply because he isn't used to being with a beautiful woman.  I was dressed very tastefully but men can't help but sneak a glance.  It may not be out of the ordinary for the lady while it may be a unique experience to get a few stares your way.

Warmly,

Emily