Whilst it is true that animal cells are more like our own, I'm not sure that would be the reason for non-allergic responses. If we take xenografts (transplanting pig organs for example) we see a profound response.
The latter theory of it having a higher mortality is more closer to the truth. Typically we observe that things we are given from childhood causes induction of tolerance (T reg cells). However this occurs best at a young age and if given orally. Peanut allergies for example are incredibly infrequent in Japan for example. Meat is given typically from a young age.
However what about the vegetarian converts or exposure to new foods as you grow older. Well the theory suggested for this are some antigens act to signal the immune system. These proteins are similar to the cell signals our body uses to tell the body there's a fight on, we are being invaded by something dangerous. This causes an immune system reaction to build up, specfically IgE antibodies. Thus on subsequent exposure, the response is stronger due to a preformed response.
Although these are they hypotheses, they're being actively investigated because frankly they can't be that simple. And of course there is conflicting evidence.
Interestingly, I've met a person with mushroom allergy. It does occur. Maybe polymorphisms (slight genetic differences) means someone responds to antigens others don't?