A healthy heart and cardiovascular system distributes oxygenated blood more efficiently than a less healthy one. Therefore, for a given level of exertion, a healthy heart will need to pump fewer times than a less healthy one to provide the same quantity of oxygenated blood to the rest of the body.
It's pretty much that simple.
However, the whole patient has to be considered. A low heart rate in and of itself doesn't mean someone is fit since there are many drugs and medical conditions that can create a low heart rate. Person A sitting in front of you appears fit and healthy, is on no relevant medications, and has a heart rate of 50? Fitness almost certainly explains it. Person B sitting in front of you is overweight, on several meds and not very fit? Fitness almost certainly doesn't explain it.