I have never heard this story, but I have only been involved in the hobby just over two years so some of the more experienced guys may be able to add more light to this.

My feeling is that many of these urban legends started out with a small bit of truth to them, but over time get embellished to the point the story totally changes.

I can think of a couple possible situations where a story like this might have begun:

1.  There are many high end neighborhoods in the greater Orlando area.  Statistically, I am sure the owners of at least one of these houses is involved in the swinging community and may have hosted some private parties at the house.

2.  I know a guy whose wife was the personal travel agent about fifteen years ago for one of the (very famous) pro athletes who live here.  Every week she was flying in "girl friends" from all over the country for the parties this guy was throwing at his huge multi million dollar home for himself and his friends.

In either of these scenarios, I can see how as neighbors might tend to talk to one another  the "brothel" story could have come into being.

Like everyone else, I am curious about this now.

...more of a swingers club. There are a couple well known ones in Orlando.