The process of sleep seems to be very disadvantageous to an organism as it is extremely vulnerable to predation for several hours at a time. Why is sleep necessary in so many animals? What advantage did it give the individuals that evolved to have it as an adaptation? When and how did it likely occur in the evolutionary path of animals?
This good non-scholarly article covers some of the usual advantages (rest/regeneration).
One of the research papers they mentioned (they linked to press release) was Conservation of Sleep: Insights from Non-Mammalian Model Systems by John E. Zimmerman, Ph.D.; Trends Neurosci. 2008 July; 31(7): 371–376. Published online 2008 June 5. doi: 10.1016/j.tins.2008.05.001; NIHMSID: NIHMS230885. To quote from the press release:
Because the time of lethargus coincides with a time in the round worms’ life cycle when synaptic changes occur in the nervous system, they propose that sleep is a state required for nervous system plasticity. In other words, in order for the nervous system to grow and change, there must be down time of active behavior. Other researchers at Penn have shown that, in mammals, synaptic changes occur during sleep and that deprivation of sleep results in a disruption of these synaptic changes.
I found this paper by Benington and Heller that expands on the previously mentioned theory of sleep as a mechanism to renew metabolism. They hypothesise that sleep is necessary to replenish glycogen stores (mainly within astrocytes) in the brain. These stores are normally used to supplement blood glucose due to the high energy demands of the brain.
It is suggested that this may also result in the manifestation of feeling sleepy as a consequence of the exhaustion of glycogen supplies in specific small areas of the brain. Brief and localised depletion of glycogen stores mean that cells are operating with less energy than they normally have to work with. This causes an increase in synthesis of adenosine from the breakdown of AMP. The paper maintains that the increased levels of adenosine are detected by adenosine receptors which then triggers or increases (as seen on EEG scans) the feeling of the need for sleep:
In NREM sleep (which may lend some support to the theory as NREM sleep accounts for 80% of sleep and is most physiologically different to waking), this glycogen is most efficiently replaced. During NREM sleep the release of glycogenolysis inducing neurotransmitters is reduced, allowing glycogen-synthase to predominate and glycogen levels to be restored. However, these same neurotransmitters are key in the processing sensory stimuli (by tonically depolarising neurons in the sensory cortex). Therefore glycogen replenishment will always be associated with a (strongly) reduced response to stimuli.
This leads to finally answering the question as to why sleep has an evolutionary advantage, which I will quote verbatim to maintain the authors momentum:
Glycogen replenishment during waking would be maladaptive because it would impair the organism's ability to to process and respond to sensory stimuli. Sleep has therefore evolved as a state where animals retreat to a safe environment, behaviour is suppressed and glycogen stores are replenished.
From what I've learned, there are two theories to answer that question:
Restoration - the body needs to rest in order to renew its metabolism (if an animal is active 24/7, it will constantly use up a lot energy and metabolism). It has been shown that mice that have gone a while without sleeping have a compromised immune system.
Preservation - sleep is assumed to confer a survival advantage. Night time is a dangerous time, so sleep sort of 'forces' an animal to lay low for a few hours.
Why sleep persists is fairly easy, why it is needed is an unknown.
Sleep appears to be necessary in any organism with a brain, that is anything with any kind of concentration of neurons. That is when denied it said organisms die. So all that has to happen is that the benefits of a brain outweighs the cost of sleep.
The length of sleep needed correlates with brain size, at least the REM part of sleep other parts correlate with metabolic rates. Now this can be seemingly confounded in larger complex brains (birds and mammals especially) when organisms start folding the brain to increase neuron density without increasing overall size. In this case these organisms are increasing hte "size" of the brain without making the brain larger, by increasing density. Worse some organisms "sleep" for long periods, but only a short portion of that time involves the neural activity associated with sleep (like REM), sleep in more complex animals (those with very large complex brains) contains many functions.
In organisms with tiny brains and slow metabolisms (aka the earliest things with brains) sleep does not take very long so the cost is minimal, the benefits of a brain (and thus learning) can be high. Later as brains get larger the cost goes up but so does the benefit, if it was possible for a brain to be retained without the need for sleep it should have been selected for at this stage. So in all likelihood the need for sleep is something fundamental to how neurons function and can't be changed without seriously disruptioin their function. It is not uncommon for unfavorable things to get locked in evolutionarily in this way, the cost to change them (in this case the loss of brain function) is far larger than the cost of sleep so selection keeps it around.
Now the complexity of sleep makes sense, if you already have this required period of downtime, it makes sense evolutionarily to tack on anything else that would be best done during said time. Better to use the triggers and time for existing downtime activities for any others that get added then have even more downtime. So now we have a slew of confounding factors that muddy studies and make it hard to tell what parts are essential.
We don't know why sleep is necessary, there are many ideas but not a lot of evidence. Given the complexity of sleep this is not surprising, teasing out which functions are fundamental is not easy. There is a current leaning towards it being necessary to clear out metabolites that disrupt neural function, it appears this process is highly disruptive to the brain if the brain is awake, to the point simply shutting down brain activity (movement in particular) is far safer for the organism. But as with all research this is very preliminary and sleep is poorly understood so acceptance is and should be very tentative.