Among animals the ability to control ear muscles is pretty common, but only a few humans have the ability. Is it because we in general lost this ability ?
Yes, humans have lost the ability to move their ears toward sounds like many other animals such as cats and rabbits.
It is thought that our ancestors lost the ability to move their ears around 30 million years ago when
Ear size decreased and the associated musculature changed 1 (Steven Hackley).
A review from Steven Hackley outlines the evidence that the neural circuit remains in modern human brains. Apparently, the neural circuit still responds to directional sounds and may cause our ears to curl slightly (2 - 3 mm).
This adaptation would have very little pressure on gene pool selection, as it provides little to no advantage other than slight, likely negligible, hearing improvement.
As survival fitness in terms of hunter-gatherer qualities, such as those of keen hearing and sight, are today meaningless in terms of reproductive viability, this phenotype would not persist over time, in my opinion.