If I understand evolutionary biology correctly, mammals first evolved on land as small, rodent-like creatures, in a time when reptiles were dominant on land. Eventually, they diversified into the species we know today. Some of these species, however, are aquatic - whales and dolphins readily spring to mind. Why did the ancestors of whales and dolphins leave the land to go to the oceans? How did they evolve from their original form to their superficially ichthyoid appearance today?
I'll focus on whales and dolphins (cetaceans) as you mention them by name and they are representative for other marine mammals such as seals or manatees. The evolution of cetaceans was one of the fascinating evolutionary mysteries. Clearly, they were mammals, but which mammals were their closest relatives? Clues to solve this mystery began to appear in the 1980s and 1990s.
During the 1980s and into the 1990s, paleontologists discovered several fossils that were clearly land-based mammals but shared many skeletal features with cetaceans. These fossils, including Pakicetus [(Gingerich et al. 1983)] and Rodhocetus (Gingerich et al. 1994), shared skeletal features that are unique to whales and to the Artiodactyla, commonly known as the even-toed mammals. Artiodactyls includes deer, sheep, pigs, bison, and hippopotamuses. Artiodactyls and the fossil whales share a unique skeletal feature of the ankle called an astrogalus, shown in the picture below. The astragalus is a "double pulley" structure that contributes to the ankle joint. This picture is taken from an excellent summary of whale evolution, provided by the Understanding Evolution site hosted by University of California at Berkeley. The astragalus is not present in other mammals.
The fossil whales and modern cetaceans also share unique features of the skull around the area of the ear, called a tympanic bulla. Although all mammals have bullae (plural of bulla), the bullae of cetaceans and the whale fossils is unique compared to other mammals. For more on the bulla, see this Talk Origins page. More on fossil whales can be found at the Berkeley Comparative Museum of Paleontology. The many fossils that have been found are transitional forms that link artiodactyls and modern cetaceans.
Around the same time as the fossil discoveries, genetic analysis added additional evidence supporting a relationship between the artiodactyls and cetaceans (Milinkovitch et al. 1993). Although early genetic analytical techniques were not as robust as they are today, many genetic studies have since supported and refined the conclusions of Milinkovitch et al. Current evidence shows that the closest living relative of cetaceans are the hippos (see, for example Price et al. 2005 and Agnarsson and May-Collado 2008). The relationships are shown below in this figure from the Agnarsson and May-Collado paper. The dark lines are the artiodactyls and cetaceans, formally called the Cetartiodactyla.
The question remains: why? The most likely explanation is that cetaceans evolved to exploit an unfilled ecological niche or adapted to new niches that formed as a result of plate tectonics or other types of environmental changes that occurred 50-55 million years ago. The niche describes all of the living and non-living resources needed by an organism to survive. Although land-based mammals were increasing in diversity, few or none were present in the oceans. The basic hypothesis is that the early whale-like artiodactyls, like Indohyus and Pakicetus were land-based (terrestrial) mammals that spent most of their time near the water's edge. Over time, they adapted to the niches in the ocean. Fossils like Ambulcetus and Rodhocetus showed clear evidence of swimming ability, with flattened tails and the enlarged rear feet. In addition, the nostrils shifted from the front of the face to the top of the head, which we recognize as the blowhole.
The shift to the aquatic habitat allowed these species to exploit resources that were not available to land-based mammals, thereby reducing competition for the resources. Reduced competition allows more individuals to survive and reproduce.
Similar scenarios are very likely for other marine mammals, such as seals or manatees. They evolved to take advantage of ecological niches that were not filled by other organisms. This basic concept, evolving to fill available niches, is a common outcome of the evolutionary process.
The of adaptation of cetaceans and other mammals to the oceans may be similar to that of the hippopotamus. Hippos spend most of their time in the water, and they show many adaptations that allow them to live in the aquatic environment. The eyes and nostrils of the hippo are high on the head, which allows them to remain almost entirely submerged but still see and smell, as shown below.
(Hippo photo by Johannes Lunberg, Flickr Creative Commons.)
Hippos feed underwaters, they are heavy enough to walk on the bottom of the river, and the mate and give birth underwater. The young can suckle underwater. Clearly, hippos seem to be another mammal that is "returning to water." Similar types of processes must have occurred in cetaceans for them to adapt to the marine habitat.
Agnarsson, I. and L.J. May-Collado. 2008. The phylogeny of Cetartiodactyla: The importance of dense taxon sampling, missing data, and the remarkable promise of cytochrome b to provide reliable species-level phylogenies. Molecular Phylogenetics and Evolution 48: 964-985.
Gingerich, P.D. et al. 1983. Origin of whales in epicontinental remnant seas: New evidence from the early Eocene of Pakistan. Science 220: 403-406.
Gingerich, P.D. et al. 1994. New whale from the Eocene of Pakistan and the origin of cetacean swimming. Nature 368: 844-847.
Milinkovitch, M.C. et al. 1993. Revised phylogeny of whales suggested by mitochondrial ribosomal DNA sequences. Nature 361: 346-348.
Price, S.A. 2005. A complete phylogeny of the whales, dolphins and even-toed hoofed mammals (Cetartiodactyla). Biological Reviews 80: 445-473.
$\begingroup$ +1 - Exactly. Because the seas were there, full of food. It's a great example of how evolution works. $\endgroup$ Sep 24, 2014 at 4:05
2$\begingroup$ Nice write up! I can see how a deep wader like a hippo might have started getting fins and going further out into the water, much the way manatees and sea lions seem to be doing. The phylogenetic tree really helps. $\endgroup$– shigetaSep 24, 2014 at 4:59
2$\begingroup$ @shigeta Thanks for reminding me about the hippos. I meant to include them in my answer. It was late though and I got sleepy, so went to bed instead. :) $\endgroup$ Sep 24, 2014 at 9:46
1$\begingroup$ While "filling niches" always sounds nice, it must mean that the niche is there, i.e. that no other already existing species is living there. Given that water based animals are probably more likely to adapt to niches in water I think there must be another advantage that mammals had over those that tried to filled the niece from water. Maybe something with changing water levels(tide)? $\endgroup$– PlasmaHHSep 24, 2014 at 12:35
1$\begingroup$ @PlasmaHH You're correct that the cetaceans could have adapted to a newly forming niche. It is also possible that the niche was there and empty. For example, the plesiosaurs and ichthyosaurs may have filled the niches. Their extinction would have left the niches empty. It is not safe to say that existing aquatic organisms were more likely to fill those niches, given the nature of the evolutionary process. Besides, the niches were filled first by reptiles. $\endgroup$ Sep 24, 2014 at 12:51
How did they evolve from their original form to their superficially ichthyoid appearance today?
This is an example of convergent evolution. Fish appear as they do (streamlined body shape, wide tail, fins, etc.) since these are adaptations to the underwater environment they're living and evolving in. These features are only "ichthyoid" or "fishy" because the fish 'got there first'. Instead, we can think of such features as "ocean-favourable".
Reptiles evolved on land, and hence have a generally different appearance from fish, but those which went back into the oceans to become the ichthyosaurs adapted again to live in that environment. Since they faced similar problems as fish (drag from the water, the need to maneuver in 3D, etc.), natural selection would favour the same "ocean-favourable" features as the fish (but not exclusively; for example, turtles have a rather different appearance, which is "ocean-favourable" but not as "fishy").
When mammals ventured into the oceans, they also faced these same problems and the same selection pressures, so they also gained a more "ocean-favourable" appearance.
$\begingroup$ Great answer. The idea of convergent evolution makes a lot of sense, and it's a pretty cool mechanism. $\endgroup$ Sep 24, 2014 at 23:24