Here is the link where urologists are warning men not to take hot water baths.
Why does it affect only males' fertility? Why not females also?
A mature human sperm has some mitochondria, a nucleus, an acrosome, and a flagellar apparatus. That's it. Because of the lack of other organelles in the sperm, they have to be kept in a kind of 'cold storage' so that they are viable for at least some time (usually a couple of days). The ideal temperature would be 1.8 °C to 2.5 °C below normal body temperature, i.e. 37.6 °C or 98 °F. If a man takes frequent hot water baths, his sperm count becomes low as the mature sperms get dessicated quickly.
The reason that women don't require this strict temperature regulation is that the mature ovum has the complete cellular machinery at its disposal- mitochondria, nucleus, golgi bodies, ER, the works. So it can remain viable for a much longer time (about seven days after ovulation) and has higher temperature tolerance than sperms.
Well, the answer by @An J has already explained the prima facie of your question which is: sperms need lower temperature for development. But I will be explaining in my answer: Why do the sperms actually need a lower temperature for their development?
This is a debatable subject, but there have been several models or hypothesis about this. I found this literature on it, and it summarises the thing excellently. Hope it helps:
Throughout the Cenozoic, the fitness benefits of the scrotum in placental mammals presumably outweighed the fitness costs through damage, yet a definitive hypothesis for its evolution remains elusive. Here, I present an hypothesis (Endothermic Pulses Hypothesis) which argues that the evolution of the scrotum was driven by Cenozoic pulses in endothermy, that is, increases in normothermic body temperature...
The model argues that stabilizing selection maintained an optimum temperature for spermatogenesis and sperm storage throughout the Cenozoic at the lower plesiomorphic levels of body temperature that prevailed in ancestral mammals for at least 163 million years. Evolutionary stasis may have been driven by reduced rates of germ‐cell mutations at lower body temperatures...
The fitness advantages of an optimum temperature of spermatogenesis outweighed the potential costs of testes externalization and paved the way for the evolution of the scrotum. The scrotum evolved within several hundred thousand years of the K‐Pg extinction, probably associated initially with the evolution of cursoriality, and arguably facilitated mid‐ and late Cenozoic metabolic adaptations to factors such as climate, flight in bats and sociality in primates.
About the function of the scrotum from the article
The scrotum is a sac‐like thermoregulatory structure that houses the testes at a temperature lower than the core body temperature (Tb; Moore, 1926; Wislocki, 1933; Ruibal, 1957; Setchell, 1998). The cremasteric muscles in the scrotum contract and relax to draw the testes closer to or allow them to dangle further away from the body in order to maintain the testes at an ‘optimal’ temperature for spermatogenesis (sensu Moore, 1926; Tsperm) and sperm storage of 34–35 °C (Setchell, 1998; Gallup et al., 2009; Mawyer et al., 2012). In humans, the scrotal temperature is maintained about 2.7 °C lower than the Tb (Momen et al., 2010). Notwithstanding the relative fitness benefits of a cooled epididymis vs. the cooled testis, four nonmutually exclusive hypotheses currently dominate explanations for the evolution of the scrotum: the Cool Spermatogenesis Hypothesis (Moore, 1926), the Galloping Hypothesis (Frey, 1991), the Mutation Hypothesis (Short, 1997) and the Activation Hypothesis (Gallup et al., 2009).
Cool Spermatogenesis Hypothesis
...argued that Tsperm and the maintenance of sperm viability during storage in the epididymis is about 34–35 °C (Moore, 1926; Appell et al., 1977). Scrotal temperatures that approach those of the core Tbcompromise fertility (Moore, 1926; Bedford, 1978b, 2004; Setchell, 1998), particularly if evaporative cooling of the scrotum is impaired (Momen et al., 2010).
The Galloping Hypothesis
... (Frey, 1991)proposes a trade‐off between testes vulnerability (fitness cost) and the avoidance of strong intra‐abdominal pressure fluctuations during galloping which impair spermatogenesis in abdominal testes (fitness benefit).
The Mutation Hypothesis
... maintains that the testis is a ‘hot spot’ for germ‐cell mutations and that the lower temperatures of the scrotum reduce the rates of mutation on the Y chromosome through mutagenic metabolites (Short, 1997).
The Activation Hypothesis
... storage of sperm at a lowered temperature ensures that they undergo ‘thermal shock’ during ejaculation into the higher temperatures of the female, which increases their motility and hence the probability of a successful insemination (Gallup et al., 2009).
To keep sperm viable in natural body conditions, they are kept at a relative lower temperature than the body temperature, and this can achieved by the scrotum. As sperm is produced in testis, and testis are situated in the scrotum outside the main part of the body, a pouch like structure. This arrangement is going to help to achieve a slightly lower temperature relative to the body temperature. Definitely, frequent hot water bath will lower the sperm count by hampering spermatogenesis as it disturbs the normal physiological conditions inside the testis by increasing temperature.
Another side, in females, the body temperature is not going to affect the oogenesis (ovum formation process). Instead of that you will see an increase in basal body temperature (about 0.5 °C) during ovulation (release of ovum). That means a hot water bath isn't an issue for female fertility.