Most animals are not afraid of the vet. When I was in practice, many dogs and cats waltzed in perfectly happily.
Of those that were afraid, I presume that many picked up expectations from their owner. The owners who tell themselves "My dog is going to be afraid of the vet!" will be tense and nervous themselves, and of course their pet will detect this and react the same way. Many owners were far more distressed than their pets were, expecting some huge negative reaction to the vaccination or something (the pet often didn't even notice the injection, until the owner squealed or shrieked with anxiety).
Finally, animals that experienced something negative on a previous visit -- perhaps a bad interaction with another animal in the waiting room, or their owner being afraid of needles, or actually having some unpleasant procedure -- will of course remember this.