Shouldn't be an issue love, lap away and enjoy yourself

...none of the ladies have ever complained. Some stated they liked it.

Look at this this way...

If you use some conditioner on the beard and mustache, you have a two way French tickler.  The ladies should love it.

is from this morning's breakfast, then don't do it. And why can't you dine with just your lips and tongue? DATY is not quantum physics.

What they don't like is stubble.
For years of my adult life I have had a beard. For years of my adult life I have had a bare face. Never had complaints during DATY with either. But just a tiny amount of stubble, is a whole other story

Hell no, don't refrain from enjoying intimate pleasures with that facial hair. Are you crazy?
Facial hair is sexy as hell on a man,,,and feels incredible on the skin.  (my skin that is)
In fact I much prefer & require facial hair...over soft shaven faces!

 
And it's an added perk when your kissing her down there........don't however, continuously rub your face on her tender sweet spot...as it will rub her (lips raw) not a good feeling...and then if you get smacked by her for rubbing her to hard with your goatee....she won't have any sympathyforthedevil.  
Otherwise..........go down and enjoy her nectar......gently...
 

Seems to be in style but think of yourself licking a Chia pet...for us it's in reverse.