For example, does the mutation need to occur in the DNA of either the specific sperm or the specific egg that gave rise to you?
To give rise to heritable genetic variation in the sense that it's a mutation that exists in all of your cells and is transmitted (with some probability) to your offspring, yes, it must come from the gametes, with the caveat that it need not occur specifically in the gamete, but in any of the cells giving rise to that gamete in either parent. I suppose it could also occur in the fertilized egg cell, and mutations shortly thereafter may produce a chimera where a substantial fraction, but not all cells carry that mutation, but as mutations are most common during cell division I think it's okay to simplify a bit and presume the source is the gametes for most heritable mutations.
Estimates of how many of these mutations exist per human fertilized egg (relative to their parent genomes) vary, but you can assume that each human has something on the order of dozens in coding regions that their parents lack, plus others in other areas of the genome. A whole human organism has far, far more mutations, and these cause issues with aging and cancer in their cells, but these are not typically heritable.
The number of mutations per replication depends on genome size and also the stringency of that organism's replication machinery. In general, complex eukaryotes have stronger systems for correcting replication errors, so you can expect bacteria to have higher rates of mutation. Some related reading:
Correlation between genome size and mutation rate?
How much variation in mutation rate in there in the human genome?
Bacterial division and mutation rate
Mutation rate in viruses
Veltman, J. A., & Brunner, H. G. (2012). De novo mutations in human genetic disease. Nature Reviews Genetics, 13(8), 565-575.
Each generation (per individual), approximately 74 de novo single-nucleotide variants (SNVs), three novel indels (small insertions or deletions) and 0.02 larger copy number variants (CNVs) arise in our genome
Ségurel, L., Wyman, M. J., & Przeworski, M. (2014). Determinants of mutation rate variation in the human germline. Annu Rev Genomics Hum Genet, 15(1), 47-70.