Can you get a reference from agency providers? Or is it from the agency itself/should I ask them or the specific provider? Just wondering how that works. Thanks.

As most of the time  the ref comes, if they'll even give it, from the agency owner or booker, , not the actual provider you saw.

I had a bad experience when I accepted a reference from an agency when I first started. I was told he was safe and fine but he showed up clearly under the influence of drugs and was extremely paranoid and attracting attention. I had to ask him to leave. So I am very wary about agency references.

very true!
I rather have a reference from a independent lady than from an agency.