There is simply no variation in the 16S rRNA gene between different species/strains that it would be useful to tell apart for a more accurate analysis.
The amount of sequence variation depends on the region of the 16S rRNA gene amplified, and the choice of region depends on the desired universality of your primers as well as the read length constraints of the sequencing platform. Put simply, choice of target for 16S amplicon sequencing is a tradeoff between breadth and specificity. Typically, 16S sequencing connotes the use of primers targeting the V3, V4, and/or V5 regions of the 16S gene. Such primers are often optimized for taxonomic breadth (i.e. to differentiate as many organisms as possible) and designed to create amplicons that are readily sequenced on an Illumina platform.1 Thus, strain-level differentiation is limited in vanilla 16S analyses, though use of a multi-region framework can greatly increase phylogenetic resolution.2
There is some variation that would be useful, but the noise in the current sequencing methods and the capacity of the current computational analytic techniques are not able to capture it.
Modern short-read sequencing platforms (primarily Illumina) are considered quite robust in terms of sequencing error rates. However, for amplicon sequencing techniques like 16S, the per-base error rate might exceed the expected relative abundance of low-abundance organisms, making their detection and differentiation tricky.3
There is some variation that would be useful, but it is not clear how this correlates with the phenotype it would predict therefore scientists prefer to use shotgun sequencing to investigate causal relationships between genome and phenotype.
You've identified the major benefit of metagenomic / shotgun sequencing over 16S amplicon sequencing. When you are trying to extrapolate microbiome functions from 16S data, you are constrained by the existing phenotype-to-phylogeny mappings that are present in your databases. With databases of sufficient size, such analyses can give very good estimates of microbiome function.4,5 That said, strains that are closely related at the 16S level may have very different functional profiles due the inherent flexibility of bacterial genomes. Increasingly, pan-genomics is revealing that the so-called "peripheral genome" carries genes of consequential function, and the carriage of such genes in a particular strain cannot be detected by 16S amplicon sequencing alone.6
- Caporaso JG, Lauber CL, Walters WA, Berg-Lyons D, Lozupone CA, Turnbaugh PJ, Fierer N, Knight R. Global patterns of 16S rRNA diversity at a depth of millions of sequences per sample. Proc Natl Acad Sci U S A. 2011 Mar 15;108 Suppl 1(Suppl 1):4516-22.
- Fuks G, Elgart M, Amir A, Zeisel A, Turnbaugh PJ, Soen Y, Shental N. Combining 16S rRNA gene variable regions enables high-resolution microbial community profiling. Microbiome. 2018 Jan 26;6(1):17.
- Schirmer M, Ijaz UZ, D'Amore R, Hall N, Sloan WT, Quince C. Insight into biases and sequencing errors for amplicon sequencing with the Illumina MiSeq platform. Nucleic Acids Res. 2015 Mar 31;43(6):e37.
- Langille MG, Zaneveld J, Caporaso JG, McDonald D, Knights D, Reyes JA, Clemente JC, Burkepile DE, Vega Thurber RL, Knight R, Beiko RG, Huttenhower C. Predictive functional profiling of microbial communities using 16S rRNA marker gene sequences. Nat Biotechnol. 2013 Sep;31(9):814-21.
- Wemheuer F, Taylor JA, Daniel R, Johnston E, Meinicke P, Thomas T, Wemheuer B. Tax4Fun2: prediction of habitat-specific functional profiles and functional redundancy based on 16S rRNA gene sequences. Environ Microbiome. 2020 May 18;15(1):11.
- Gordienko EN, Kazanov MD, Gelfand MS. Evolution of pan-genomes of Escherichia coli, Shigella spp., and Salmonella enterica. J Bacteriol. 2013 Jun;195(12):2786-92.