You guessed it correctly: seed dispersal by zoochory, specially when the animal eats the fruit to spread the seeds (endozoochory), is an intimate relationship between the plant and the animal: the fruit has to be attractive to that specific animal, not attractive to you or me.
And, as you also seem to know, the same fruit will taste differently for different species. We could explain that using the concep of qualia (see my answer here), but the case here is even simpler: different species may have different receptors and different sensory perceptions (not sensations, I actually mean perception) for the same fruit. The most famous case is that one involving birds and chilli peppers, which don't taste hot to them (birds lack capsaicin receptors).
All that being said, olive (Olea europeae) fruits attract birds, which are their main dispersal agent (REY et al., 1997; Spennemann and Allen, 2000).
According to Spennemann and Allen, 2000:
Worldwide, olives have proved to be a successful invader of disturbed lands, with birds as the principal seed vectors.
So, we can hypothesise that olives don't taste as bitter to birds... or, that birds do like a bitter taste. It really doesn't matter: what's important is that the fruit is eaten and the seed is dispersed.
A male blackcap (Sylvia) eating an olive. Source
- REY, P., GUTIERREZ, J., ALCANTARA, J. and VALERA, F. (1997). Fruit size in wild olives: implications for avian seed dispersal. Functional Ecology, 11(5), pp.611-618.
- Spennemann, D. and Allen, L. (2000). The Avian Dispersal of Olives Olea europaea: Implications for Australia. Emu - Austral Ornithology, 100(4), pp.264-273.