Here's how it went:
"So when am I going to get your number?" - him
"Uhh... its right here in the rolodex. You can find it anytime" - me
"I know I can find your number there, but I'm asking you for it. That's different."
"hmm... "
"I'm not scaring you am I?"
"Uh... no. Here's my cash drop. See ya."

At the end of the day, I gave it to him. I'm not sorry I did. :)