I don't want to add much more about that. Since childhood I noticed a chilling effect, even when doing rollercoaster-type activities.
Probably because the blood would be forced, physically, that is, by inertia to move upwards during free-fall, thus causing the genital's blood supply to become obstructed thus reducing their temperature and thus leading to the sensation of them being frozen as you described. Plus you might feel cooler all over your body due to the air flow against it and since your genitals wouldn't be as biologically favoured when your body's survival instincts for low-temperature environments kicks in you're body would probably reduce the blood flow to your genitals and other extremities favouring your vital organs such as your heart, lungs, brain, etc. with a greater blood supply to keep them warm and working.