This is a really great question. To take a specific example, let's consider Amanita phalloides, the death cap mushroom. It doesn't advertise its poison; in fact, it resembles several edible species of mushrooms, including Caesar's mushroom and the straw mushroom. It contains alpha-amanitin, an extremely potent poison that inhibits RNA polymerase, shutting down protein synthesis. The estimated minimum lethal dose is 0.1 mg/kg or 7 mg of toxin in adult humans. The poison has a delayed effect, taking between 10-24 hours for significant effects to appear, so it's not like a poisoned animal can learn to avoid the mushroom in the future. None of the amatoxins appear to have any purpose except as a mammalian poison. They aren't specific to humans, and are capable of poisoning dogs. So what's the evolutionary advantage of this poison?
According to an essay written on the California Native Plant Society blog by John Chesnut, a horticulturist, the answer might lie in where the mushroom grows -- on tree roots as ectomycorrhiza (emphasis mine):
Death cap is an ectomycorrhizal symbiont. This means it forms connections on the root-tips of forest trees; in California, its typical (but not exclusive) partner is coast live oak. [...] Ectomycorrhizal (EC) fungi collect major nutrients, nitrogen and phosphorous, and exchange these with the host tree for sugars. Delicate hyphal strands extend outward from the root tip mass into the surrounding soil and mulch. EC also allows efficient active transfer of macronutrients, micronutrients, and soil water to the tree. The chronic phosphorous limitation in serpentine soils makes the EC symbiosis especially important for local forest types on this soil. Studies in Norway discovered up to 50% of a birch tree’s sugar is exchanged at the root tips with EC symbionts.
So the poison might not be intended as a way of preventing itself from being eaten, but as way of defending the tree it lives with from nematode pests (emphasis mine):
So why are Amanita so poisonous? It is an unlikely deterrence to vertebrate predation of the fruiting caps, as the effect is slow-acting (36-72 hours before the toxic crisis in humans) and the toxin is not concentrated in the cap. Evidence supports the hypothesis that the fitness obtained from synthesizing the toxin is secured within the hyphal network. Perhaps toxic Amanita obtain nitrogen from poisoned nematodes, or protect themselves (and their symbiont hosts) from plant parasitic nematode predation.
Perhaps the toxin suppresses the growth of competing fungal webs. It seems clear the toxic effect of death cap is intrinsic to its invasive success worldwide.
Interestingly, he points to some research suggesting that some mushroom flies might have co-opted this mechanism to keep themselves parasite-free (link and emphasis mine):
An evolutionary entomologist working in New York State, John Jaenike, has discovered four species of mushroom flies in the genus Drosophila that lay eggs in the gills of fruiting Amanita phalloides. The fruit fly taxa are related to ones that inhabit rotting skunk cabbage, but in New England have recently transferred to the recently introduced Amanita fruitings.
Jaenike discovered that Amanita phalloides is toxic to the damaging parasitic nematodes Howardula that reproduce in the stomach of fruit flies. The toxicity of the death cap to the parasitic nematodes results in much greater egglaying (fecundity) by the fruit flies. The fruit flies are affected by the toxic amanitin, especially the males, but the poison is more than offset by the increase in reproduction.
Janike also discovered that most other insects using mushrooms as egg laying sites (craneflies and forest gnats) shun use of the Amanita (due to its toxicity).