The one and only time I have ever come face to face with LE was at a very upscale appartment with very discreet entrances.  Why?  NOSY NEIGHBORS!   One of her neighbors complained to the landlord who called the local police and asked them to investigate.  The neighbor was asked to call them the next time they noticed "suspicious visitors".  I guess having different gentlemen knock on your door a few times a day is suspicious enough for some busybodies.  

And I have heard several accounts of other providers having this type of issue in the past as well.  The other thing you have to be concerned about is that some guys, especially those who become one of your regulars, may have trouble distinguishing fantasy from reality and start to think its not a service and these guys turn into stalkers eventually when you have to explain the boundaries to them.  Do you really want a disgruntled former client to know where you live?  

Its best to keep your personal life completely sheltered from this business.