Your question is very sensible; we would indeed expect this kind of negative frequency-dependent selection: i.e., if toxin-producing plants are common, non-toxin-producing plants should gain an advantage because they don't pay the costs of producing toxins but they do (or may) gain an advantage from herbivore deterrence. (Conversely, if toxin-producing plants are rare, they could gain an individual-level benefit by being the only ones in the population to deter herbivores.)
As of 2007, however, it seems that the empirical evidence for this phenomenon was pretty weak. In the following, "resistant genotypes" means genotypes that have some kind of anti-herbivore trait (typically production of secondary compounds that deter herbivory) [emphasis added]:
... Only a few studies have examined the role of frequency-dependent selection on plant defenses against herbivory (i.e., Berenbaum & Zangerl 1998, Roy 1998, Siemens & Roy 2005). ... No study has manipulated the frequency of resistant/tolerant host genotypes explicitly to examine if the evolution of plant defensive strategies is subject to frequency-dependent selection. However, some studies tested some of the predictions of the frequency-dependence selection hypothesis (Berenbaum & Zangerl 1998, Roy 1998).
The study by Berenbaum and Zangerl (1998) found that in Pastinaca sativa, the range of variation in the level of a resistance component (coumarin) against its consumer Depressaria pastinacella did not change from 1873 to present, suggesting a cyclical dynamic promoted by frequency-dependent selection. In a reciprocal transplant experiment Roy (1998) did not find evidence that common hosts are eaten more and have lower fitness than rare hosts. In a recent paper, Siemens and Roy (2005) evaluated the following: (a) if the amount of damage experienced by the most common host genotype was higher than that expected by its frequency, and the amount of damage experienced by the most rare host genotypes was lower than that expected by its frequency (assortative damage hypothesis); (b) if the most common host
genotype had higher levels of damage and lower average fitness compared with the more rare host genotypes; and (c) if the amount of damage correlated positively with the frequency of the most common host genotype. ... Unfortunately, their results provide little support for the hypothesis of frequency-dependent selection acting on plant defenses against herbivory.
... Although two studies have explored the potential for
frequency-dependent selection on plant resistance, no study has yet
examined this possibility on plant tolerance. Before
frequency-dependent selection can be put aside as an explanation of
the maintenance of variation on tolerance and resistance, explicit
experimental tests of the hypothesis are required.
(I haven't followed up the more recent literature to see if there is more recent evidence ...)
There is a related phenomenon that's much better documented: toxin-producing bacteria produce antibiotics to kill other bacteria of their own and other species, to minimize competition for resources. In "rock-paper-scissors" systems (2), cheaters (bacteria that are resistant to toxin but don't produce it themselves) invade toxin producers; then non-resistant bacteria invade the cheaters (because there's no toxin being produced any more); then toxin producers reinvade (because nobody's resistant any more, so there is once again an advantage to producing toxin).
I would say that the connection between this type of frequency-dependent selection and the selection that maintains polymorphism for the sickle-cell allele is loose. Sickle-cell is usually cited as an example of heterozygote advantage. It is loosely speaking a form of negative frequency-dependent selection at the level of alleles rather than genotypes; a gamete carrying a sickle-cell allele is more likely to end up in a heterozygote (and thus gain an advantage via malaria resistance) when it's rare and more likely to end up in a homozygote (and lose fitness due to sickle-cell disease) when it's common. It's also theoretically possible (although I haven't seen it discussed, and I would guess that it's not very important in practice) that increasing frequency of sickle-cell heterozygotes could also feed back at the population level to reduce the incidence of malaria, lower the advantage of resistance, and hence generate a more analogous form of negative frequency dependent selection.
- Núñez-Farfán et al. "The Evolution of Resistance and Tolerance to Herbivores", Annual Review of Ecology, Evolution, and Systematics, Vol. 38 (2007), pp. 541-566 http://www.jstor.org/stable/30033871
- Kirkup and Riley, "Antibiotic-mediated antagonism leads to a bacterial game of rock–paper–scissors in vivo", Nature 428, 412-414 (25 March 2004) .