Have you ever been attracted to a client where you considered a relationship outside of the business?  

I've seen it discussed occasionally here.

This path is frought with obsticles.   If you really feel this way, you need to have an adult conversation with the recipient & see if it is reciprocated and if there is suffient basis to form a civie relationship.  Go Slow.  Think this through.  Proceed with caution but I see nothing wrong with it.
Clients forming relationships with providers rarely work either but do so just often enough that we can not say "Never."
If this is not going to work, you will likely have to stop seeing him.  This also happens with more frequency.  It would be torturing yourself to keep it up.
skb