Why does the butterfly have a cocoon stage in its life cycle? It could have simply grown up within the egg/cocoon, and emerged as a butterfly. Instead it is first an egg, then a worm, then a chrysalis, and finally after some time a butterfly.
Holometaboly is believed to have evolved only once (Labandeira, 2011), but is arguably the most successful mode of development we know of in terms of species richness (Kristensen, 1999 - PDF link). Insects which have adopted this method are far more diverse than their hemimetabolous counterparts. A conservative estimate by Hammond (1992) is that holometabolous insects comprise more than 60% of all living organisms.
The subclass Pterygota (winged insects) is divided into (super orders) endopterygota (wings develop interiorly) which are holometabolous, and the exopterygota (external wing development) which are hemimetabolous.
Hemimetabolous insects (usually) develop from an egg to an adult through a series of immature forms (known as nymphs or instars) which are slightly modified (and usually slightly larger) forms of the preceding form. There are exceptions to this mode within the exopterygota including asexual parthenogenesis which is common in aphids. The general life cycle is well illustrated in the Dermaptera (earwigs). All stages inhabit the same ecological niche and so maintain the same general body plan.
Image file from Wikimedia Commons here
Although hemimetaboly is a more gradual process than holometaboly, it happens in a small number (often four or five nymphal instars) of discrete steps rather than the more continuous development of other animals such as mammals. Although all insects fall into either the exopterygote or endopterygote categories (i.e. wings either develop externally or internally), there are examples of intermediate processes which lie somewhere between hemi- and holometaboly. A notable example are the Cicadas. Cicadas are Hemiperans (true bugs) which lie in the exopterygota and so are, by default hemimetabolous. However there is a considerable difference between their immature and mature stages:
Illustration by Debbie Hadley, using drawings from Insects - Their Ways and Means of Living by Robert Evans Snodgrass, U.S. Bureau of Entomology. These drawings are in the public domain. From here.
During the immature stages, the larva (or nymph) will burrow underground, sometimes for many years. This requires strong front appendages, which they also use when they surface to grip onto a surface whilst they shed their final skin to become an adult. This differential niche exploitation is likely to have given rise to the morphological differences between immature and mature life stages.
Exploiting different niches during different developmental stages not only reduces intra-specific competition between young and adult, it also enables the insect to be more specialised to fulfill their specific role, which changes as they develop. The role of the immature is to grow, they do not yet need to be able to reproduce or disperse; whereas the role of the adult is to find a mate and reproduce. Young often develop below ground where there are fewer predators, adults often fly to enhance dispersal (further reducing competition and also decreasing inbreeding). These two activities call for vastly different body designs.
Some other common holometabolous insect orders are Coleoptera (beetles), Hymenoptera (bees, wasps, and ants), and Diptera (true flies), whereas hemimetabolous groups include Orthoptera (grasshoppers, crickets etc.), Mantodea (praying mantids), and Blattaria (cockroaches).
As for the chrysalis which Lepidoptera (butterflies and moths) have evolved to surround their pupal stage, the main role is likely to be protection. Larvae (caterpillars) are a highly nutritious and energetic food source and thus have adapted defensive mechanisms such as mimicry, warning colours, and dropping to the ground when disturbed. This drop would damage the metamorphosing pupa (the most vulnerable stage as it is usually relatively immobile), as the cuticle softens to allow for cell-rearrangement.
The process of insect development is regulated by juvenile hormone (JH), see Konopova et al, (2011) for more on this.
Hammond, P. (1992) Species Inventory. Global Biodiversity (ed B. Groombridge), pp. 17–39. Chapman and Hall, London.
Konopova, B., Smykal, V. & Jindra, M. (2011) Common and Distinct Roles of Juvenile Hormone Signaling Genes in Metamorphosis of Holometabolous and Hemimetabolous Insects. PLoS ONE, 6, e28728.
Kristensen, N.P. (1999) Phylogeny of endopterygote insects, the most successful lineage of living organisms. European Journal of Entomology, 96, 237–253.
Labandeira, C.C. (2011) Evidence for an Earliest Late Carboniferous Divergence Time and the Early Larval Ecology and Diversification of Major Holometabola Lineages. Entomologica Americana, 117, 9–21.
The cocoon phase allows lepidopterans (and other holometabolous insects) to reorganize their body and undertake the dramatic metamorphosis from larvae to adult in a safe environment. Cocoons and pupae are typically attached to branches by silk (oftentimes after climbing high into the canopy), thereby removing them from many predatory interactions and providing a protected environment to breakdown the larval body and develop into the adult phase.
In my mind, the main adaptive significance of the cocoon/pupal phase is to provide an environment within which the animal can metamorphose while reducing exposure to both predators and any harsh elements.