There are several ways to infer into the propagation delays of auditory system in vivo that is applicable to humans (i.e. leaves the subjects intact).
One of them is so-called ABR that stands for Auditory Brainstem Response. As it is clear from its name this method allows you to derive the brainstem response to an auditory stimulus (so, you can track the auditory pathway up to its input into thalamus), but its resolution is so fine that you can separate single components ("waves") for the following path segments (from Wikipedia):
Wave I – generated by the peripheral portion of cranial nerve VIII
Wave II – generated by the central portion of cranial nerve VIII
Wave III – generated by the cochlear nucleus
Wave IV – generated by the superior olivary complex/lateral lemniscus
Wave V – generated by the lateral lemniscus/inferior colliculus
So, it takes only 10 ms for the signal to reach the input into thalamus. This correlates well with some auditory signal based activities, like startle reflex, which was also shown to have about 10 ms latency.
I tried to search for some electrophysiological recordings from auditory cortex (temporal lobe) paired with auditory stimuli, but the first paper I came accross that is available for free (and which I can also link from my answer) is this work done on monkeys.
If you look at Figure 1:
This is the results from multiunit recordings and you see that the initial activity in the cortex starts approximately 20 ms after the stimulus onset with a short activity pattern, followed by some short break and then longer period of activity with broader frequences spectrum.
In my opinion the delays in signal propagation are not due to complex signal processing in the olivary nucleus or any other brainstem structures, rather they are due to electro-chemical transformations in CNS synapses.
The initial arrival of the signal into the cortex does not mean the perception of the sound, it definitely takes hundreds of millisecond until the conditioned cortex-mediated response to this stimulus.