Why is there wide variation in genome size amongst groups of protists, insects, amphibians and plants, but less variation within groups of mammals and reptiles?
The quick answer is that there is no "why" in evolution; things happen and if they're beneficial they tend to stick around more than the deleterious things. The longer, (slightly) more satisfying answer is non-coding DNA. Thanks to non-coding DNA the size of a genome doesn't even correlate to the number of genes. The human genome, for example, has over 20,000 genes but is less than two percent protein-coding; the flower Utricularia gibba, however, has over 28,000 genes with about two orders of magnitude smaller of a genome, due to the fact that its genome is 97% protein-coding. In short, there is no real meaning to be found in genome size variation due to non-coding portions, which while they are sometimes important regulators can often be just transposons (40% of our genome). I will also point out that all of those groups you mention have species with significant polyploidy.
The are a number of other interesting ideas. In the process of researching this, I stumbled upon the Animal Genome Size Database, which is an absolute gem of a treasure trove. It's an unbelievable mine where I've spent a good half-hour just reading. In particular, the statistics pages list interesting correlations for certain groups. Mammalian genome size, for example, correlates:
- Positively with red blood cell size, despite lack of nuclei
- Negatively with metabolic rate
- Not at all with developmental rate or longevity
For amphibians, however, there is:
- Positive correlation with red blood cell and nucleus sizes in both frogs and salamanders
- Negative correlation with developmental rate in many groups
- Negative correlation with brain complexity in both frogs and salamanders
- No strong relationship with metabolic rate in either frogs or salamanders
- Some association with intensity and frequency of metamorphosis
If you have time, I highly recommend reading through T. Ryan Gregory's PhD thesis entitled The C-Value Enigma. It's around 350 pages, but each chapter is in separate pdfs and is the most in-depth treatment I've seen, despite being about a decade old. It's been moderately life-changing for me to read, some of the interpretations he presents challenged my thinking on the subject of genome evolution entirely. Should genomes be treated as individuals, and what would that mean? I digress. Chapters 3, 4, 6, and 7 are what you'll want, in particular the last twenty or so pages of chapter 7.
Dr. Gregory now has a faculty position and his lab's research page is a little easier to read. I love this image:
As genome size increases in eukaryotes, the percentage of the genome composed of genes (white circles) becomes smaller. Conversely, the percentage of the genome made up of transposable elements (black circles) increases with genome size.
It seems like the best way to look at genome size and find some meaning is within a specific group. There are correlations, for example, that are present in frogs and salamanders, but not in either group alone. Some of the groups are very unbalanced; in insects, only Orthoptera tend to have very large genomes. Interestingly, a number of those groups also undergo some form of metamorphosis, but the correlations aren't what you might have expected: insects that develop quickly and metamorphose have smaller genomes, but "[i]n a more general sense, small genome size seems linked to rapid diversification... Conversely, insects without metamorphosis may possess very large genomes..."