I am told that there are some species, like fish or rabbits, that if let, will eat their own children. If this is true, how does a species like this exist? Shouldn't the fact that they kill their own lineage make them nonviable?
Yes, it is true.
Prairie dogs for example are known for frequent infanticides.
Many other species kill their babies too
But of course, such behaviour also exists in other lineages such as grey langurs, gerbilles, lions, giant water bugs and Bottlenose dolphins (just to cite a few examples).
How does that evolve
It will be impossible to provide a complete universal explanation to this behaviour because the evolutionary processes causing this behaviour varies from lineage to lineage. For examples, in lions, only males kill young of the females that are still nursing and they do so when taking over a new harem only. In prairie dogs, mothers cause infanticide preferentially on others' babies but also on their own babies.
Going into the details of how such behaviour evolves in every specific lineage would probably require writing an intro on kin selection and other fields of evolutionary biology which is way too much for a single post. You may want to have a look at the wikipedia article infanticide for a start.
Shouldn't the fact that they kill their own lineage make them nonviable?
Of course, they don't kill all the babies. Only a fraction of them!