Midtown/East Side. The Financial district is great for guys who work in that area. If your client base is concentrated with that category, it is great (some lovely hotels, etc.). But, if not, you're much better staying Mid-town

But you'll probably find that more gentlemen prefer Midtown.  For your own benefit, you may want to consider that there isn't much to do in FiDi.  And after business hours, it becomes a ghost town.