A mutation that increases fertility among the predators should quickly spread in such a population
probably isn't true if there are tradeoffs involved. It also wouldn't be true if the increased death rates due to overpopulation, were concentrated more heavily on the over-breeders and their descendants than on the rest of the population. (This could happen due to spatial structure; for example, if each predator hunted and raised their offspring in their own patch of territory.)
Tradeoffs are the bread and butter of Life History Evolution. ("Life History Traits" are the traits which are most directly related to fitness: age-specific mortality rates, age-specific fecundity rates, age of maturity, etc. Size is an honorary life history trait, as is offspring size, since these often have strong effects on mortality and fecundity.)
One of the most famous examples from the early days of Life History Theory is the Lack Clutch. The ornithologist David Lack noticed that birds never laid as many eggs as they were capable of producing, and wondered why. He proposed that there was a tradeoff between the number of offspring, and the survival of each of the offspring. He suggested that clutch size was designed to maximize the number of offspring who would survive to the age of fledging, and that this would be an intermediate optimum.$^1$
Lack was right that the most fit clutch size would be intermediate. In detail, clutches are usually a little smaller than he predicted, because there are additional tradeoffs he didn't consider.
Here's a partial list of tradeoffs that are often thought to be important.
- The more offspring there are in this clutch, the less fit each of them will be. For example, maybe the parent has a certain amount of resources to invest in creating offspring; the more of them there are, the smaller each of them has to be (larger animals are often more fecund).$^2$ If the offspring fitness cost is in terms of survival, then this is Lack's hypothesis.
- The more offspring there are in this clutch, the less likely the parent is to survive to future breeding seasons. (This is often called the Cost of Reproduction.) For example, maybe the parent has some resources which it has to allocate between reproduction, and its own maintenance. The optimum will not favor perfect maintenance (this is the basis of the Disposable Soma theory of aging);$^3$ but it also favors intermediate clutches (which are a little smaller the Lack Clutch).$^4$
- The more offspring there are in this clutch, the less offspring there will be in future clutches. For example, maybe the parent has some resources which it has to allocate between reproduction now and its own continued growth (which will make it more fecund in the future). This tradeoff also has to do with the age at maturity: at what ages should an organism not reproduce at all (devoting all its resources to growth), and at what age should it start?$^5$
(Tradeoffs are often modeled as limiting-resource allocation problems, but this isn't the only sort of tradeoff. A tradeoff occurs whenever getting more of one good thing requires getting less of a different good thing [or more of a bad thing], for whatever reason.)
So much for life history tradeoffs in general. But, in the face of these tradeoffs (and whatever other constraints), which can have different strengths and different shapes etc., different species hit on different strategies. Why? What leads to the large predator life history being optimal for large predators? I don't know. I don't even know if it is known! Life History Evolution is complicated, with many different traits coevolving together. It's hard to model more than a few of these (holding the others constant) at a time. I don't know if anyone's actually explained the conditions that would lead to an entire suite of life history traits, from scratch. (Once upon a time, attempts would have been made to try and explain the large predator life history as one end of the r-selection vs. K-selection spectrum. But that paradigm has fallen out of favour among researchers.$^6$)
Here's another factor, elaborating on the first tradeoff, which I think is probably relevant.
If there are too many predators, then there won't be enough prey, and the average death rate will soon catch up. If this food-shortage cost were applied to every member of the predator population equally, then a mutation for increased fecundity would indeed be favoured: although all members shared the cost (increased mortality), only the mutant would get the benefit (increased fertility); on average the mutant would be fitter and the mutation would spread. (This is ignoring the other costs, such as slower parental growth and faster parental aging.)
But now suppose that you are a predator, and you own a patch of territory; and whether you over- or under-hunt mostly affects your territory. Suppose also that you care for your offspring, or at least let them share the hunting in your territory, rather than sending them elsewhere. Then over-breeding will lead to over-hunting in your territory (or, to hunting the same amount but having lower-quality offspring). But the other territories'll be fine! The brunt of the mortality cost of over-breeding and over-hunting, as well as the fertility benefit, will fall on you and your offspring.
tl;dr: if conditions (eg. due to the population's being spatially structured) are such that the mortality cost of higher fertility applies more strongly to the over-breeders than to the rest of the population, then there will be selection against over-breeding. In addition to costs of overpopulation, higher fertility can also negatively affect the parent's fitness in other ways; for example, by using up resources that they would have used for self-maintenance, or resources that they would have used for growth.
- Lack, "The Significance of Clutch Size". Ibis (1947).
- Smith and Fretwell, "The Optimal Balance Between Offspring Size and Number". American Naturalist (1974).
- Kirkwood and Rose, "Evolution of Senescence: Late Surival Sacrificed
for Reproduction". Philosophical Transactions B (1991).
- Charnov and Krebs, "On Clutch Size and Fitness". Ibis (1974).
- Kozlowski, "Optimal Allocation of Resources to Growth and
Reproduction: Implications for Age and Size at Maturity". Trends in
Ecology and Evolution (1992).
- Reznick, "r- and K-Selection Revisited: The Role of Population Regulation in Life-History Evolution". Ecology (2002).
The classic textbook for Life History Evolution is Stearns' The Evolution of Life Histories (1992). A more recent one (I haven't read it) is Roff's Life History Evolution (2002).