This always disturbs me when someone says these things. Many of us visit our gentlemen at their homes. The gentlemen have no reservations about our loyalty and discretion otherwise they would not be inviting us there. I dress appropriately at all times so there is never a chance anyone would have an inclination to a conclusion as to anything about me. There is always a story you can tell your neighbors to suffice their curiousity and more times than not they didn't even notice.
It is impractical for LE to arrest one person at a time so no outcalls are not usually anything to worry about.