EDIT: Thanks a lot to @abukaj for pointing out the mistake in my answer (and to @paracetamol for asking such a beautiful question). I am rewriting my answer to incorporate the (hopefully) correct background knowledge this time.
NEW ANSWER: As @paracetamol and @AlanBoyd (in their answer) pointed out the lack of credible support for the claim that infants produce rennin/chymosin, the previous answer attributing the digestion of casein in infants to chymosin had to be corrected. So, lets now look at the other possible candidates for casein (or, more specifically, $\kappa$-casein) digestion in infants.
As Martin et. al. (2016) claim, milk casein has some bioactive roles, such as forming masses with calcium and phosphorus1. Also, infant milk formulas mostly have higher casein content than human breast milk, making the former harder to digest than the latter. Back to the digestion point, human breast milk is known to contain many proteases in itself, including anionic trypsin, elastase, plasmin, cathepsin and kallikrein, while prothrombin has also been identified in human colostrums2. In fact, Ferranti et. al. (2004) also identified fragments of casein created by plasmin cleavage3. However, Chatterton et. al. (2004) were able to detect $\beta$-casein after 1 hour of in vivo gastric digestion in 8 day old infants, but not in 28 day old infants, suggesting the increase of digestive capacity of the infants4. This increase can be attributed to the development of gastric acid production system in the infants. For the case of newborns, Nakai and Li-Chan (1987) suggested that Pepsin C, also called gastricsin, can hydrolyze casein at the pH range of 4-5 (the lowest pH newborns can reach)5. However, the buffering effect of human and bovine milk proteins does not allow the gastric pH to reach the optimum pH required for activity of pepsin A and pepsin C.
How, then, is casein digested by infants at all? This is summarized (to some extent) by Chatterton et. al. (2013), according to which, although gastric pH is unable to reach the optimum pH for pepsin activity, it is sufficient for minor initial digestion of human milk proteins. This causes the release of glycomacropeptide, the C-terminal portion of casein. The combination of this initial digestion and the gastric pH, close to the pI of caseins (~ pH 4.6) causes precipitation of caseins and slowing down of their digestion. The casein peptides, on the other hand, are absorbed into plasma where they show a number of biological activities, including inflammation modulation and blood pressure regulation6.
Thus, even though infants lack chymosin and a fully mature digestive system, this is compensated by a combination of proteases in breast milk and the slower digestion process until the digestive system of the infants develops completely.
Martin CR, Ling PR, Blackburn GL. Review of Infant Feeding: Key Features of Breast Milk and Infant Formula. Nutrients. 2016;8(5):279. Published 2016 May 11. doi:10.3390/nu8050279
Dallas DC, Underwood MA, Zivkovic AM, German JB. Digestion of Protein in Premature and Term Infants. J Nutr Disord Ther. 2012;2(3):112. doi:10.4172/2161-0509.1000112
Ferranti, Pasquale & Vittoria Traisci, Maria & Picariello, Gianluca & Nasi, Antonella & Boschi, Velia & Siervo, Mario & Falconi, Claudio & Chianese, Lina & Addeo, Francesco. (2004). Casein proteolysis in human milk: Tracing the pattern of casein breakdown and the formation of potential bioactive peptides. The Journal of dairy research. 71. 74-87.
Chatterton DEW, Rasmussen JT, Heegaard CW, Sørensen ES, Petersen TE. In vitro digestion of novel milk protein ingredients for use in infant formulas: research on biological functions. Trends in Food Science & Technology. 2004;15:373–383
Nakai, S. and Li-Chan, E. (1987) "Effect of Clotting in Stomachs of Infants on Protein Digestibility of Milk," Food Structure: Vol. 6: No.
2, Article 8.
Chatterton, D. E. W., Nguyen, D. N., Bering, S. B., & Sangild, P. T. (2013). Anti-inflammatory mechanisms of bioactive milk proteins in the intestine of newborns. The International Journal of Biochemistry & Cell Biology, 45(8), 1730–1747