This paper, The biomechanics of foraging determines face length among kangaroos and their relatives might provide some insight, but very likely misses a lot of factors that would apply, e.g., to wolves vs cougars. In the case of carnivores or omnivores, hunting strategies, prey capture methods, and even tearing & chewing techniques are likely to interact synergistically with muzzle length via the Baldwin effect (Wikipedia). Another paper The Baldwin Effect and Genetic Assimilation compares the Baldwin Effect to the Waddington effect, which can have similar results.
In response to the suggestion by @Willk and @tyersome, I've added the following explanation:
The Baldwin Effect is widely accepted among evolutionary biologists. The Waddington Effect is not as widely accepted and is sometimes (incorrectly) interpreted as a form of Lamarckism.
In essence, the Baldwin Effect works like this:
- a population is subjected to a new environmental challenge or opportunity
- a few individuals learn a new strategy that overcomes the challenge or exploits the opportunity
- natural genetic variation makes some individuals in the population more capable of learning the new strategy
- Individuals whose genetic makeup makes them more capable of learning the new strategy have a fitness advantage and produce more descendants, so over time the proportion of individuals in the population who can are genetically equipped to learn and use the new strategy, increases.
- Individuals whose genetic makeup gives them a physiological advantage in regard to using the new strategy will have a further fitness advantage, and will consequently produce more descendants than others in the population.
Let's say the environmental challenge for two different kinds of carnivore (let's call them Bogs and Dats) is to catch Mophers. Both Bogs and Dats initially have the same medium-to-short muzzles. Some Bog individuals figure out that they can dig Mophers out of their burrows, and some Dat individuals figure out that they can catch Mophers at night when the Mophers leave their burrows. Both strategies are successful. Some Bogs happen to have longer muzzles than their cousins, and find it turns out that longer muzzles work synergistically with the digging strategy, allowing Bogs to stick their noses into the Mopher burrows to grab escaping Mophers. The resulting fitness advantage results in an increase of the long-muzzle trait in further generations of Bogs. Note that in this scenario it is the adaptive behavioral strategy that creates selective pressure that favors a particular genetic adaptation.
Dats on the other hand, because of their nocturnal hunting strategy, benefit from improved night vision; and long muzzles don't provide any fitness advantage to Dats because Dats don't dig Mophers from their burrows. As long as Bogs and Dats don't hybridize, they will most likely end up with long and short muzzles respectively.
The Waddington effect, also called “Genetic Assimilation”, is somewhat more direct:
- An environmental stress causes a proportion of a population to develop one or more abnormal traits, by interfering with embryological development.
- If there is a selective pressure in the environment that favors some subset of those traits, individuals whose genetic makeup makes them more likely to develop that subset of traits, those individuals are likely to produce more descendants than other members of the population.
- If being “more likely to develop” that subset of traits results from a weakening of genetically determined development controls that would otherwise prevent development of that subset of traits, then the subset of traits can eventually become the normal phenotype.
An explanation of how the Waddington effect might influence muzzle length requires a bit more hand-waving, largely because there is no evident environmental factor that would cause perturbations in the embryological development of, e.g., dogs. However, I'd assert that an abrupt release of a strong selective pressure can have a closely analogous result. Wolves really have to look and act a lot like wolves in order to survive in the wild (partly because of their survival strategies), so they are subject to strong selective pressure. Domestic dogs, on the other hand, are subject to much weaker selective pressure. Moreover, when a variant shows up (a pug, for instance which has a very short muzzle), it is often considered “cute” and is bred separately to fix the trait in a subpopulation. Although the "release of selective pressure" mechanism isn't quite what Waddington contemplated, the process is closely analogous.
Note that the Waddington effect and the Baldwin effect are not mutually exclusive. Relaxation of selective pressure that reduces fitness of variants, and environmental factors that cause disturbances in embryonic development (a la Waddington) both produce a rich source of variants upon which the Baldwin effect depends.
However, It is very difficult to prove that any specific phenotypic trait in an existing species was caused by a particular selective pressure. At best we can only make plausibility arguments supported by models and rather sparse environmental & ecological data from the geological record, from analyzing genotypes, etc. In the case of canines vs felines (which split off from a common ancestor perhaps 42 million years ago), we can really only guess what the environmental challenges & opportunities might have been, and what behavioral strategies may have been adopted by different individuals and populations.