TL;DR - It's just a scientific convention. Whether it is good, broad enough etc is another question.
The question is a bit like asking why in mathematics, the set of natural numbers does not contain negative numbers. The reason is simply because it was defined like that as it was considered a useful working definition.
As you noted the definition of a gene has radically changed, from being defined by the oberved phenotype, to a molecular definition agnostic to functional consequences.
Maybe your interpretation of the old definition of a gene (as an inheritable, modifiable piece of genetic information) is correct, but modern nomenclatures tend to focus on what one may call defining "minimal functional categories". You could define a gene as a group of all the sequences that lead to a given phenotype, sure, but for a complex phenotype like "being tall", you would have to include a huge part of the genome in your definition of the gene (growth factors, transcription factors, cellular metabolism, all of it!!). Pretty much anything could be a gene of some kind given a certain context. And if everything is a gene, well, then the definition is too broad and not really useful.
That is why a bottom-up definition is more useful: define the gene as small units that may or may not infuence certain phenotypes, and then define the phenotype as the result of all the genes involved.
Many things that can be inherited and that exist in various forms are the product of distant pieces of DNA, some coding for proteins, others for regulatory RNAs, so the scientific community decided to define the gene as the "minimal functional unit" of genetic information. That does not mean that each gene is unaffected by other factors (just like for example a "cell" is a functional unit but must be considered as a part of a broader context to understand how it works), so genes may interfere with each other, overlap etc.
It's also important to emphasize that the gene is not only the protein/RNA coding sequence, but also includes promoter regions, untranslated ends and many other regulatory elements that can also have inheritable phenotypic consequences. So pretty much anything functional is part a gene; enhancers are a special case, and a recent challenge to this contemporary definition of a gene, because they challenge the previously accepted notion that genes are structurally continuous and therefore spatially well-defined.
