My chin used to do that, too. Initially I thought she was scared, but it was out of the blue also. I would even put her back in her cage and laid my hands flat so she could jump away and she's stay on my palms and shake. It started happening after she got clawed by a cat and I figure it was actually just her being scared and freezing. She doesn't do it anymore but it took a good while for her to completely stop doing it.
Just try to go slow and maybe hold him a little less and just play with him in his cage.