Indeed, many plants produce molecules which are poisonous to things that eat that plant (insect, mammal, bird) in the "hopes" of preventing that animal from eating them. In this sense, even eating the "right" fruit may be dangerous (if that individual plant happens to be an exceptionally vigorous producer of the poison, or because the eater is weakened for some other reason, or simply ate too much). I doubt insects would eat the wrong fruit that much, because evolution favors insects that can precisely recognize correct prey plants, at least in indigenous surroundings.
Indeed, many plant "flavors" that we humans enjoy (such as garlic and peppers) are likely such herbivore poisons that we happen to be immune to. In fact, almost every plant is actually poisonous - the question is, poisonous to which species?
Plant defense is a very actively researched topic these days, since it relates to things like agricultural efficiency and global food shortages (and besides, it's a very interesting phenomenon). Here is a review discussing what you ask about in more detail: http://www.plantphysiology.org/content/121/2/325.full
An addendum upon reading this question many months later: While evolution has perfected the ability of insects to recognize correct food, this has not been the case for man-made pesticides. I would guess that the caterpillar you saw is more likely to have eaten an artificial chemical (either pesticide or pollutant) than a "wrong fruit".