I would not in any way call you paranoid. Glad you went with your gut instincts and didn't say anything or even reply. I've been doing this long enough to know that LE can do many things (not textbook) to make a collar. But they need you to say or do something that is going to stick and not get thrown out in court.
Very early on when I first got into this, I was super wet behind the ears. I got caught up in something similar but WAS NOT arrested simply because I didn't say what I guess should have been said. Before the undercover slid into the bathroom he asked me "Do you have protection?" I simply said "I keep a taser?" and laughed. When he asked me "Do you do greek? " Honestly, not even understanding what that was at the time (but knew it had something to do with sex), I replied "If you're hungry, there's a nice tavern across the street we can visit and then come back if you like". Apparently my answers weren't what they were looking for. When I got the loud knock on the door as 3 more rushed in like I had a rifle aimed out the window, made me have a seat as one checked my arm (I suppose for tracks), while asking me who I worked for. I said "no one" , while another asked "Hey, when he asked you if you did greek, why did you not answer?" I turned to him and asked "WOW, are you hungry too?" Needless to say, they had nothing. and simply my ad wasn't one that said will do this or that.
They did issue me a citation, left and eventually I left as well. I went to court a couple of months later and the Judge seemed to be more frustrated with LE. My case was dismissed and yrs later I had it expunged so it wouldn't show even a dismissal. Of course I could always say I got caught up and explain my way out, but why bother. I learned so much from that, that it changed how I did things from that point on. I'm glad it happened though, it made me more aware of the mistakes that both clients and providers make.
I will say this, he was a quick talker and hesitated on a hug or kiss much in the same manner. However, being a man he removed his shirt but nothing else as I suggested he get comfortable. So, no you are not paranoid. Cautious is more like it.