Indeed, many plants produce molecules which are poisonous to things that eat that plant (insect, mammal, bird) in the "hopes" of preventing that animal from eating them. In this sense, even eating the "right" fruit may be dangerous (if that individual plant happens to be an exceptionally vigorous producer of the poison, or because the eater is weakened for some other reason, or simply ate too much). I doubt insects would eat the wrong fruit that much, because evolution favors insects that can precisely recognize correct prey plants, at least in indigenous surroundings.
Indeed, many plant "flavors" that we humans enjoy (such as garlic and peppers) are likely such herbivore poisons that we happen to be immune to. In fact, almost every plant is actually poisonous - the question is, poisonous to which species?
Plant defense is a very actively researched topic these days, since it relates to things like agricultural efficiency and global food shortages (and besides, it's a very interesting phenomenon). Here is a review discussing what you ask about in more detail: http://www.plantphysiology.org/content/121/2/325.full