Are there any major noticeable limitations to genome sequence compression methods that use reference templates?

Question

Recently, I have been researching about big data analytics in biochemistry, and started wondering about how genome sequence compression could affect analysis.

Of all the method listed on the Wikipedia page, the reference template method is my favourite as, it not only seems effective, but also was the idea that popped into my head before I did any study regarding this topic.

But, before I implement it in a project I am working on, I wanted to know if there are any common and obvious drawbacks/limitations that the bioinformatics industry faces quite often when using this scheme.

Answer 1

The most obvious drawback of the reference template method is that if you use it, you've already analysed data.

Usually, you get data from wet biologists. They sequence samples and upload sequences to a server. Often an uploading process is already automated. If you sequence in-house, the data is already on your server. If not - biologists do not want to do an extra job. Moreover, if it is not a big project you do not have that much data that transferring and storing it is the problem. So you'll get the data as *.fastq.gz files.

Then you make QC, align reads and make an SNV-calling or expression analysis or whatever you want. Aligning and SNV-calling could be painful, complicated (Copy Number Variation, heterozygosity etc) and time-consuming. It is the problem. Resulting data usually is stored as Variant Call Format (VCF). Under the influence of CRAM and 1K Genome Project which uses it, VCF has been developed.

Answer 2

The biggest drawback is that reference template cannot address the question of genome sequence compression. It addresses variant information compression.

Granted, a lot of the interest in genomics has been addressed towards variants, basically because they are easier to analyze than actual whole genomes. Additionally, when many people say "genomics" or "genome sequence" what they mean is "human genomic variation", basically because there is more money to be made with human genomics than with other genomes; and therefore most people are really talking about human genomic variants when they talk about "genomes".

Even in the variant case, templating approaches stop working very well when you have structural variants of the genome, or at least they become extremely complex. More recent advances like genome graphs can address some of the biggest issues, such as structural variants, but they still have not very well defined (AFAIK) limits in terms of how divergent the genomes can be before they break down.

But leaving that aside, any two sufficiently divergent genomes cannot be compressed by the reference template method or by graphs. The number of differences becomes so large that it is ludicrous to record the differences between the genomes, as that information can end up being larger than just recording the two full genomes and compressing them independently. This becomes clear if you consider trying to compress a Drosophila and a Saccharomyces genome together, or even very closely related genome sequences such as human and chimpanzee, which have: "...approximately thirty-five million single-nucleotide changes, five million insertion/deletion events, and various chromosomal rearrangements."

At that degree of divergence, it is much easier and almost certainly more space conscious to just separately represent the two genome sequences and compress them independently. For a review of tools in this space, you can see here.