It doesn't matter if its hand or any other human body organ, cold would hurt, the reason being that we humans being warm blooded animal require an optimal temperature-pressure balance to be maintained for our nervous system to work in its natural order.
So when we have, in your case, cold hands its like a thermal shock to our hand as the above mentioned functional optimality is hampered, and just similar to a hot ceramic cracking due to cold water poured on it, our hand(would not crack, however might in extreme cases) reacts to this with the sensation termed as Pain.
This is a common answer to your questions, cold is directly proportional to pain being felt during our daily engagements(like holding a bag).
Pressure plays a vital role in this too as when your hand is cold the pressure exerted to the epidermis is at a reduced rate, which is another reason for the same.
Hence commonly my conclusion would be the Temperature-Pressure Factor.