I am super attracted to an older man who exudes refinement, sensitivity, and an ability to relax and enjoy life's blessings. If a lady on your arm is presents herself in a classy manner (nails clean, hair done, modest yet form fitting dress, good jewelry never hurts) then the glances you get will be looks of jealousy - not skepticism. It isn't that rare of a combination to have an older man and a younger woman.
However, if the lady looks disheveled and can't take care of herself - then it's just an eyesore and people will notice. Choose carefully! There are plenty of great women out there.
Also, another note is that a previous gentleman once told me that he was uncomfortable with the amount of looks simply because he isn't used to being with a beautiful woman. I was dressed very tastefully but men can't help but sneak a glance. It may not be out of the ordinary for the lady while it may be a unique experience to get a few stares your way.