I doubt anyone knows the answer to which aspect of variation jays are considering when sampling nuts (although my guess would be mass). Langen (1999) did do experiments that explored the sampling behaviour you've noticed — they offered jays samples of peanuts with ranges as wide as 1 g (i.e. over a range from 1 to 2 g). The analysis mostly focused on when jays would be choosy (lots of nuts vs few nuts available; low vs high variability; competitive vs non-competitive scenarios; subordinate vs. dominant individuals; etc.)
Jays were reasonably accurate at visual assessment (the first nuts they sampled were statistically significantly heavier than a random sample of the available nuts), and their subsequent sampling behaviour (what you're observing) further improves their success in selecting heavier-than-average nuts:
Jays improve their accuracy at selecting a high-quality peanut via the initial visual scan and the subsequent sampling behavior. Peanut sampling (using the beak to handle several peanuts in succession) results in a higher rate of food storage than is attainable by simple visual inspection, at least when options are not too similar.
In order to figure out what aspect of the nuts birds are actually assessing, you'd have to do experiments where you found a way to independently vary the number of nuts, mass, sound of the nut when shaken, etc., and see what happened.
Jablonski et al. (2015) actually did experiments with Mexican jays (Aphelocoma ultramarina) to evaluate proximal mechanisms of nut choice, and suggested that they do use auditory cues — but the authors don't actually show any tests of auditory cues:
(abstract) Jays evaluated each peanut by performing fast movements of the head combined with additional fast movements of the beak, which may open and subsequently close producing sound at the moment of hitting the shell. These movements seemed to provide Jays with additional sensory information that led to a more strict discrimination against non-preferred peanuts. ...
(discussion) We are now analyzing a series of experiments that seem to indicate that sound is indeed taken into account by Jays handling the peanuts and that it may be more important than the perception of heaviness ...
If these subsequent analyses were ever published, I haven't been able to find them online ... if you were really interested you could contact the authors and ask them ...
For what it's worth, there is also a paper titled "New Caledonian crows infer the weight of objects from observing their movements in a breeze" ...
Again, my guess would that the jays are 'weighing' the nuts. Mass should be pretty easy for them to assess, and closely related to the energetic/nutritional reward given by a particular nut.
Langen, Tom A. “How Western Scrub-Jays (Aphelocoma californica) Select a Nut: Effects of the Number of Options, Variation in Nut Size, and Social Competition among Foragers.” Animal Cognition 2, no. 4 (December 3, 1999): 223–33. https://doi.org/10.1007/s100710050043.
Jablonski, Piotr G., Sang-im Lee, Elzbieta Fuszara, Maciej Fuszara, Choongwon Jeong, and Won Young Lee. “Proximate Mechanisms of Detecting Nut Properties in a Wild Population of Mexican Jays (Aphelocoma ultramarina).” Journal of Ornithology 156, no. 1 (December 1, 2015): 163–72. https://doi.org/10.1007/s10336-015-1193-6.
