I make it 8%. Here is my reasoning.
The gene is X-linked.
40% mutant males, so freq(mutant allele) = p = 0.4, and freq(wt allele) = q = 0.6
To get a mutant female we have to have a mutant male parent, probability = 0.4
Of these matings one half will produce a female offspring so 0.4*0.5 = 0.2
i.e. 20% of matings derive from a mutant male and produce a female offspring.
Now look at the female mate:
probability(mutant) = p2 = 0.16
probability(carrier) = 2pq = 0.48
probability(wt) = q2 = 0.36
so our 20% of matings that have the potential to produce a mutant female offspring partition as:
mating with a mutant female: 0.2 x 0.16 = 0.032 mutant female offspring
mating with a carrier female: 0.2 x 0.48 = 0.096 of which:
0.048 mutant female offspring
0.048 carrier female offspring
mating with wt female: 0.2 x 0.36 = 0.072 carrier female offspring
(sanity check - 0.032 + 0.096 + 0.072 = 0.2)
Thus the probability of random mating producing a mutant female is 0.032 + 0.048 = 0.08 (8%)
Incidentally there is another way of thinking about this. Note that the H-W frequency of mutant females in the population is 16%. One of the assumptions of H-W is random mating. So the probability of a random mating producing a mutant female = p(female) * p(mutant if female) = 0.5 x 0.16 = 0.08
So - where have I gone wrong?