The frequency-selectivity of loudness perception was first shown in the 1930s, when Fletcher and Munson published a set of curves showing the ear's sensitivity to loudness compared to frequency. These equal-loudness curves, as shown in the question above, are now known as Fletcher-Munson equal-loudness contours.
One straightforward physical explanation is the fact that sounds with frequencies around 4 kHz are amplified in the middle ear canal through the process of resonance. In effect, they are perceived 10 to 20 dB louder than other audible frequencies. The pinna also affect resonance and hence the outer and middle ear together join forces to create humps and bumps in the equal-loudness curves.
The evolutionary reason why 4 kHz, and not, e.g. 0.5 or 8 kHz, is difficult to pinpoint. First I can clarify some misconceptions in the question, before getting back to the question proper:
- Firstly, you mention that the "characteristic frequency" of a human voice is between 80 and 250 Hz. You are referring to the fundamental frequency; the characteristic frequency is a measure used in electrophysiology to determine the frequency-specificity of inner-ear neural structures.
- Secondly, the fundamental frequency does not determine the frequency-selectivity of speech understanding. A useful picture is given below in Fig. 1, which shows which frequencies convey which speech components.

Fig 1. Frequency dependence of speech components. source: The Essential Guide to Hearing Loss
It becomes apparent from Fig. 1 that consonants (f, s, h etc.) are higher pitched than vowels (a, o, u etc.). Arguably, consonants convey most of the word information. Indeed, speech understanding is generally less affected by removing the vowels than by removing the consonants. One may deduce from all this that high-frequency hearing (say 2 - 8 kHz) is most important to understanding speech.
And then, to get back to your evolutionary motivation behind the Fletcher-Munson equal-loudness contours, it may, purely hypothetically be so, that our ancestors with a middle-ear resonance around 4 kHz were advantaged over those that had resonances elsewhere in the audible frequency range, if any, because of an enhanced perception of the spoken word. But this is entirely and purely speculative and remains an unreferenced statement for which I do not wish to be held responsible in any way.