contact the agency and make sure they are OK with it

I always check the provider's information, to make sure the reference is legit.  
 
I'm not sure if this provider gave you her personal number, or if she's an independent only loosely affiliated with an agency. But a reference's contact info needs to be verifiable business information. Meaning that, if I google her number... a few of her ads should pop up, and that number should also be listed on her website, reviews, p411, etc. Otherwise, I'll have no way to know that the person I'm speaking to is actually a legitimate provider (meaning, that you could be trying to fool me).  
 
FYI:  
 
Some independents will ask for more than one reference, as is their prerogative.  
 
I personally don't accept agency references unless they are also corroborated by an independent. Some girls don't accept agency references at all.  
 
For me, the main reason is that the person giving the reference is usually an agency owner who has never met you face-to-face. I want to speak with someone who has actually spent time with you.  
 
Also, having worked for agencies, I know that they're (generally) more about quantity than quality. It's about getting as many people in and out the door as possible. So if a guy's hygiene is not great, or he's a total jerk, the agency usually continues to book him... even if the girls complain. On the other hand, as an independent, I care a lot about how I'm treated... and not just the major stuff (safety, getting paid), I care about the subtle things as well. Will my boundaries and time be respected? Are you pleasant company? Do you have good hygiene? Etc.  

I'd never see most of the clients I met through an agency.
 
However, as I said, if you do have an additional reference from a reputable independent (and everyone swears you're a nice guy), then you're good... That's just my method, though. Other women may work differently. Just don't expect one reference to be a golden ticket... especially if it's from an agency