As I understand it, various animal traits have to evolve gradually, but what happens to the species that are "neither here nor there"?
To put it differently, if a species evolved from another, it did so because it's somehow better, right? So why are there examples of the original species not being extinct?
What factors determine weather some species "stick"?
Mathematician/computer programmer's answer here:
There is a continuum of different animals — in fact it's pretty fair to say that every animal occupies a different place on this continuum. They're just not uniformly distributed over the continuum; they're clustered around forms that are most likely to survive and reproduce, and the lowest-energy paths between them.
This is because evolution is basically a stochastic optimization algorithm, one that finds the "best" set of parameters for maximizing some function by randomly perturbing an initial set of input values. In fact, some of the best optimization algorithms today are based on ideas drawn directly from evolution and called "evolutionary algorithms".
In mathematics, given enough time, these algorithms will all converge on one optimum solution and nothing else. Why isn't it like that in nature? Because in mathematics, the "fitness function" that we're optimizing for stays the same for a given problem for as long as the algorithm runs. It represents the specific problem we're trying to solve. In nature, there's no outside force imposing a fitness function; an organism's survival depends on millions of factors in its environment which change over time, many of which depend on the survival and properties of the other organisms in its environment (competition, symbiosis, predator/prey relationships, etc.) This is a chaotic system so complex that it could easily go for billions of years without reaching a steady state, and even if it did, external changes (like the odd asteroid impact, to use an extreme example) would still come around to shake things up.
Another reason for the clustering is because a lot of the "intermediate states" in the genetic space have a fitness of zero — these are the states between species that can't interbreed, or that have sterile offspring. The categorization of "species" is based on this, and although it's not exact, it's still generally true that the more different two creatures are, the less likely they can have viable offspring. This makes evolution more likely to explore the spaces near already-successful organisms, and less likely to produce radical new things by hybridization.
Short answer
Why are there species rather than a long continuum?
Three important reasons I could think of are sex, non-uniform adaptive landscape and ancestry.
Long answer
I am not sure I'll answer your question so let me know if I miss your point or if I help!
To start with, you might want to read this answer on the semantic difficulties behind the concept of species
What factors determine whether some species "stick"?
Natural selection is nothing but differential fitness (fitness is a measure of both reproductive success and survival) among genotypes within a population. Individuals having greater fitness will leave more offsprings and therefore the genes of these individuals increase in frequency in the population. There are few generalities to be made about what phenotypic traits are beneficial in a given population. For example, "white fur" is a very good trait for a polar bear but would highly deleterious for a mealworm.
However, there is a thing called species selection wherein a given lineage at least, it is possible to identify specific traits that seem to either reduce the extinction rate or increase the speciation rate. This is, for example, the case for polyploidy in angiosperms (Whitton and Otto, 2000)
if a species evolved from another, it did so because it's somehow better, right?
If you observe different extant species you cannot say that any of these species evolve from any other one you can today observe. The correct way of looking at two species is that they share a common ancestor in a given past. Therefore, looking at a cat and a blue tit you cannot say that one species evolved from the other one but you can only say that these two species share a common ancestor (just like any other pair of species) that was neither a cat nor a blue tit. The example is obvious because cats and blue tits are "not so closely related" (everything is relative) but the same logic holds for any pair of species.
Why are there species rather than a long continuum?
Sex
The simplest and most obvious reason why there are species within which individuals are more similar compared to each than to individuals from other species is due to the definition (the most common definition because different definitions exist!) itself of a species. A species is a group of individuals that can interbreed. See this for more info on the concept of species.
Take two originally different groups of individuals and allow them to interbreed. Their traits will mix up and you won't be able to tell two different groups apart. All individuals within the new mixed group are a mixture of the individuals from the two previous groups (under some circumstances this process has been sometimes called "reverse speciation"). If now you take one single group of individuals. You split them into two groups in the sense that you don't allow individuals from group 1 to mate with individuals from group 2. You will see that after some evolutionary time, the individuals of group 1 will tend to resemble much more to individuals of group 1 (its own group) than to individuals of group 2. If you wait long enough so that these two groups of individuals become different enough so that they can't interbreed any more because they diverged too much, then you have what is called a reproductive isolation and under the common definition of species, you can say that a speciation (You may want to have a look to the wiki article for "speciation") occurred and therefore you have two new species instead of one ancestral species.
why the two groups tend to diverge through times?
You may wonder "But why the two groups tend to diverge through times?". There are several processes that explain that divergence:
If you are not very familiar with these concepts I recommend that you have a look at Understanding Evolution (UC Berkeley).
Adaptive landscape
Note also that there are other reasons for explaining this pattern. One other reason is "Because the adaptive landscape is not a flat function". What this means to the layman is that there are some combinations of traits that cannot really be beneficial.
Ancestry
Also, individual phenotypes are not independent of each other and not only for ecological reasons but also because of shared ancestry. If you consider two families, you will easily accept no to see a continuum of phenotypes but two distinct groups (maybe in one family curly hair is common while in the other they all have straight hair).
Nothing happens to them. Organisms exist. They breed with other organisms who are genetically compatible. We humans might try to categorize them according to certain traits, but our labels are just labels, biology isn't governed by them.
Over time, we might see that a population used to have one trait, and its descendants no longer have it, they look different. Nothing earth shattering happened, no bright line was crossed, there was just a change in allele frequencies.
if a species evolved from another, it did so because it's somehow better, right?
No. This is just plain wrong on your part.
It's just different. Maybe it changed so that its traits better match the current environment, or maybe the change was random drift. You can't easily categorize one species as "better" than another.
Typically when both new and old species still exist it is because evolution pushed the new one into a different habitat or role.
As a hypothetical example reef fish vs. deep water fish and their relative size. Let's say deep water fish evolved into reef fish, but we still have deep water fish. So there were deep water fish that were a little smaller than the rest of the deep water fish, and this gave them access to a new place to hide from sharks, shallow waters near reefs. As time goes on this puts evolutionary pressure on the fish to shrink so as to better hide in the reef, those "neither here nor there" fish may have gotten some benefit from being near the reef but the smaller fish got even more benefit and eventually outcompeted the middle species. Vice versa for the deep water fish vs this middle species. It was not as good in deep water so it was outcompeted there as well. This continues until evolution has separated them into two new species.
edit:
What factors determine whether some species "stick"?
Evolution optimizes for the current environment, as long as that environment is stable and the species is a good fit for it then there is little pressure to change. If the environment changes then a species will adapt to it. Here environment is everything relevant to the species: predators, food availability, weather, everything that impacts their life.
Organisms on Earth did not evolve in a homogenous environment. A critical part of speciation (when you go from a single species into two or more) is a reproductive barrier.
This can be a literal, physical barrier - mountain range appears between two populations, valley in the middle of habitat floods and isolates the two halves of the population, a small group is thrown by some catastrophe on a remote island and cannot escape, etc.
It can also be a genetic barrier: Imagine a bird species where males compete for mates with their bright blue crests, and rare mutations occasionally lead to red-crested males which cannot mate at all. If some female birds happen to mate with undesirable red-crested males for a few generations, two parallel sub-populations may develop: Birds which prefer red crests and bird which prefer blue crests. These populations may be very unlikely to interbreed.
There may be more complicated ways of erecting a barrier. For instance, a population of flowering plants that could previously interbreed arbitrarily may find that the ecosystem has experienced some crisis, and now the pollinating insects have become fastidious and only visit certain flowers and not others. Another example: While humans are currently a single species, because of culture (eg. language) certain human subpopulations (such as European nations) are much more likely to breed within themselves than between themselves.
Regardless of how the barrier comes to be, once a barrier can separate a species into sub-populations, the machinery of speciation is set in motion. All species evolve over time in different ways, especially if their environment does not have a very long history of unusual stability. As populations evolve, they try to stay somewhat coherent - the changes tend to be such that they still permit everyone in the population to mate with each other; otherwise they would impose a fitness cost.
However, if two subpopulations are not in contact, there is nothing enforcing compatibility between them. Therefore, as evolution does its work, these are free to wildly diverge from each other. Recall the example of the bird species in which males with blue crests enjoy reproductive success. The color itself is not particularly important, but it is important that the males all have the same color crest and the females prefer the same color. So as these birds evolve, the crest color can slowly drift in hue.
Now let's say you took these birds, and set a few of them free in one continent, and another group free on another continent. Again, over time, the crest color will shift. However, there is nothing stopping from the color in continent A from shifting to red while the color in continent B shifts to green. There is, after all, no advantage to being compatible with a population you are not in contact with.
The example above is largely behavioral, but non-behavioral examples are also possible. A very fundamental process is fertilization: Eggs have an ECM made up of proteins unique to that species, while sperm have enzymes to digest the coat of their own species. Because of this, cross-species fertilization is very difficult. Again, once you erect some kind of barrier between two populations, the systems of coat proteins and enzymes in the gametes of either population may evolve in divergent ways - they evolve in small steps, so that the interaction partner protein can always keep up, but compatibility with the isolated population is not selected for, and if the isolated groups are reunited after a very long time their gametes may end up becoming unable to fertilize each other.
I shall introduce you to the phenomena called ring species
Examples include gulls, Ensatina salamanders, house mouse, etc
A ring species is a series of adjacent populations that can interbreed with each other. Say population A, B, C, D, E.
A can interbreed with B
B can interbreed with A and C
C can interbreed with B an D
D can interbreed with E
All very good. Sounds like a regular species. But here is the weird part. The populations furthest from each other cannot interbreed. ie Population A cannot interbreed with Population E.
A and E are by definition different species. And yet, population A and E are linked, genes can flow between A and E via adjacent populations.
What is most likely happening is that there is a gradual change in gene variation between populations, so that there is gradually increasing barrier between successful hybridization. This barrier increases until the terminal ends of the ring cannot hybridize.
Furthermore, if any intermediate population within the ring were to go extinct, and break the ring, the two ends of the ring would become two separate species.
A ring species is a snapshot of one species that is one local extinction event away from becoming two species.
