What room are you letting him out in? I know if I let Chinchy out in the bedroom, which is fairly big, he'll avoid me because he knows I'm the one who puts him back in his cage and ruins his fun. If you let him out in the bathroom though, he'll jump all over me because it's much smaller. I think being in a smaller space like that might facilitate bonding. Also, chins tend to prefer "scritches" (similar to the way you'd scratch a cat under their chin, behind the ears, etc). A good way to go about giving scritches (in my experience anyway) is to ask permission before you do it, by putting your hand palmside up in front of them, asking them softly if it's ok, and then scritching underneath the chin. If you scritch on their head, behind the ears or whatever, they may feel threatened and think that you're trying to pick them up.
Also, most chinchillas don't like being held. They aren't generally very cuddly animals, so I wouldn't expect your chin to like it. Hopefully these suggestions help though.