I have had a lot of interaction with LE over my lifetime, and no I am not white, yet I have still been able to find a balance between allowing my rights to be trampled without being confrontational with LE.
My response upon said partner stating something along the lines of "Why do you ask? Do you have something to hide?" would be something along the lines of "officer I can assure you my business is quite law abiding, but as I am not accustomed to having to account for my day to day business, I would like to ask you again, Am I a suspect in a crime?
If the partner tries to escalate, I would simply add "by your tone I can only assume that you suspect me of doing something illegal, in which case I don't see any benefit to me in speaking to you without the benefit of counsel. Is there anything else I can do for you good gentleman? and can I be on my way? or should I be placing a call to my attorney?"
Nothing inflammatory, nothing argumentative, and more importantly, no false statements that can bite me in the ass.
This too is just my experience, but with the many encounters that I have had with LE in my life, it's been over thirty years since I was slammed on the hood of my car. The key is "polite, but firm" LE knows that polite, well spoken, individuals with resources are the most likely to be believed by Judges, DA's and watch commanders. They will almost always go find a "softer" target.