The biggest drawback is that reference template cannot address the question of genome sequence compression. It addresses variant information compression.
Granted, a lot of the interest in genomics has been addressed towards variants, basically because they are easier to analyze than actual whole genomes. Additionally, when many people say "genomics" or "genome sequence" what they mean is "human genomic variation", basically because there is more money to be made with human genomics than with other genomes; and therefore most people are really talking about human genomic variants when they talk about "genomes".
Even in the variant case, templating approaches stop working very well when you have structural variants of the genome, or at least they become extremely complex. More recent advances like genome graphs can address some of the biggest issues, such as structural variants, but they still have not very well defined (AFAIK) limits in terms of how divergent the genomes can be before they break down.
But leaving that aside, any two sufficiently divergent genomes cannot be compressed by the reference template method or by graphs. The number of differences becomes so large that it is ludicrous to record the differences between the genomes, as that information can end up being larger than just recording the two full genomes and compressing them independently. This becomes clear if you consider trying to compress a Drosophila and a Saccharomyces genome together, or even very closely related genome sequences such as human and chimpanzee, which have: "...approximately thirty-five million single-nucleotide changes, five million insertion/deletion events, and various chromosomal rearrangements."
At that degree of divergence, it is much easier and almost certainly more space conscious to just separately represent the two genome sequences and compress them independently. For a review of tools in this space, you can see here.