Some cats, which are separated from their home, have the ability to travel back to their original home - even over long distances and land that they haven't encountered before. E.g., see the Time article on the mystery of the geolocating cat or the following blog entry.
Based on the material above there seem to have only been some suspected mechanisms a few years ago.
It is explained with path integration in behavioral neuroscience. Not only cats, but other mammals, birds and even insects use path integration to return to a starting point.
Here is a relevant excerpt from the book Beyond the Cognitive Map: From Place Cells to Episodic Memory (by A. David Redish):
Path integration is the ability to return directly to a starting point (sometimes called a home base or reference point) from any location in an environment, even in the dark or after a long circuitous route (Barlow, 1964; Gallistel, 1990; Maurer and Seguinot, 1995). Sometimes called dead reckoning, this ability has been shown in gerbils (Mittelstaedt and Mittelstaedt, 1980; Mittelstaedt and Glasauer, 1991), hamsters (Etienne, 1987, 1992; Chapuis and Scardigli, 1993), house mice (Alyan and Jander, 1994), rats (Tolman, 1948; Alyan et al., 1997; Whishaw and Maaswinkel, 1997), birds (Mittelstaedt and Mittelstaedt, 1982; von Saint Paul, 1982), and even insects (Wehner and Srinivasan, 1981) and arthropods (Mittelstaedt, 1983), as well as dogs, cats, and humans (Beritashvili, 1965).
Path integration in animals has been the subject of argument for more than a century, including a notable debate in 1873 between Alfred Wallace and Charles Darwin in which Wallace suggested that animals find their way back via sequences of smells and Darwin argued that animals must be using dead reckoning (see Wallace, 1873a, 1873b; Darwin, 1873a, 1873b; Nature, 1873; Forde, 1873; Murphy, 1873). The carefully controlled experiments of Mittelstaedt and Mittelstaedt (1980) and Etienne (1987) have demonstrated conclusively that this ability is a consequence of integrating internal cues from vestibular signals and motor efferent copy.
emphasis mine
Cats, dogs and rodents can use taxon navigation as well along with path integration. Here is a relevant excerpt from Neural compass or epiphenomenon? Experimental and theoretical investigations into the rodent head direction system (by Matthijs van der Meer):
“Internal allocentric” navigation. This type of navigation relies on a mapping of praxic commands to an allocentric spatial representation, allowing the animal to make a direct return to a home base after a complex outward path in the absence of cues, an ability referred to as path integration, discussed in detail in the next section. Returning by following an odour trail back, or navigating to a cue indicating the home base would be taxon navigation, but behavioural experiments have provided convincing evidence that rodents are able to do this without using external cues. As mentioned above, this ability requires some mechanism of continuously updating at least one’s directional heading (a “homing vector”) relative to the home base. This is an allocentric representation which does not need to be related to any external cues, unlike the next class of strategies; the reference point or direction can in principle be set to any location the animal desires.
emphasis mine
Further reading:
