It is explained with path integration in behavioral neuroscience. Not only cats, but other mammals, birds and even insects use path integration to return to a starting point.
Here is a relevant excerpt from the book Beyond the Cognitive Map: From Place Cells to Episodic Memory (by A. David Redish):
Path integration is the ability to return directly to a starting
point (sometimes called a home base or reference point) from
any location in an environment, even in the dark or after a
long circuitous route (Barlow, 1964; Gallistel, 1990; Maurer and
Seguinot, 1995). Sometimes called dead reckoning, this ability
has been shown in gerbils (Mittelstaedt and Mittelstaedt, 1980;
Mittelstaedt and Glasauer, 1991), hamsters (Etienne, 1987, 1992;
Chapuis and Scardigli, 1993), house mice (Alyan and Jander,
1994), rats (Tolman, 1948; Alyan et al., 1997; Whishaw and
Maaswinkel, 1997), birds (Mittelstaedt and Mittelstaedt, 1982;
von Saint Paul, 1982), and even insects (Wehner and Srinivasan,
1981) and arthropods (Mittelstaedt, 1983), as well as dogs, cats,
and humans (Beritashvili, 1965).
Path integration in animals has been the subject of argument
for more than a century, including a notable debate in 1873 between
Alfred Wallace and Charles Darwin in which Wallace suggested
that animals find their way back via sequences of smells
and Darwin argued that animals must be using dead reckoning
(see Wallace, 1873a, 1873b; Darwin, 1873a, 1873b; Nature, 1873;
Forde, 1873; Murphy, 1873). The carefully controlled experiments
of Mittelstaedt and Mittelstaedt (1980) and Etienne (1987) have
demonstrated conclusively that this ability is a consequence of integrating
internal cues from vestibular signals and motor efferent
Cats, dogs and rodents can use taxon navigation as well along with path integration. Here is a relevant excerpt from Neural compass or epiphenomenon?
Experimental and theoretical investigations into the
rodent head direction system (by Matthijs van der Meer):
“Internal allocentric” navigation. This type of navigation relies on a mapping of praxic commands to an allocentric spatial representation, allowing the animal to make a direct return to a home base after a complex outward path in the absence of cues, an ability referred to as path integration, discussed in detail in the next section. Returning by following an odour trail back, or navigating to a cue indicating the home base would be taxon navigation, but behavioural experiments have provided convincing evidence that rodents are able
to do this without using external cues. As mentioned above, this ability requires some mechanism of continuously updating at least one’s directional heading (a “homing vector”) relative to the home base. This is an allocentric representation which does not need to be related to any external cues, unlike the next class of strategies; the reference point or direction can in principle be set to any location the animal desires.