3
$\begingroup$

As someone working with sequencing data and sequence files I was wondering whether is there a compression method / format for sequence / read files? Of course one can use general compressors like rar or zip to compress a fasta file, which is suitable for transferring data, but reading the sequence out of it is infeasible. Also I'm aware of BAM file format which is a block copressed gzip file of a SAM file, but as far as I know it is mainly used in alignments. But is there a compression for plain fastq(-like) files for storing millions of reads or a collection of sequences while being able to get sequence and quality info directly from the file?

$\endgroup$
  • $\begingroup$ Normally one gzips such things. Do you actually need random access and, if so, how many reads/sequences are we talking about? BTW, BAM isn't a bunch of block gzipped SAM files, it's its own binary format that's block gzipped. $\endgroup$ – Devon Ryan Mar 24 '15 at 22:02
  • $\begingroup$ Yes, I know I just didn't express myself well I'll edit my question, thx for pointing that out $\endgroup$ – Nandor Poka Mar 24 '15 at 22:55
  • 1
    $\begingroup$ Ohh I forgot to answer your question in my previous comment. I'm talking about millions of reads or tens of gigabytes of data. $\endgroup$ – Nandor Poka Mar 24 '15 at 23:04
  • $\begingroup$ Sequence files can be compressed after Burrows-Wheeler transformation; but you cannot read them directly. See here. Bowtie indices are burrows wheeler compressed; you can use bowtie-build to transform your fasta file. $\endgroup$ – WYSIWYG Mar 25 '15 at 5:36
  • 1
    $\begingroup$ Note that block compression can be applied to any type of file, including fastq or fasta, using the bgzip utility that comes with samtools. That said, I don't know to what point this makes the file more easy to deal with than just using plain gzip. Biopython has some degree of support for bgzip. $\endgroup$ – bli Mar 25 '15 at 10:27
3
$\begingroup$

BAM is the current standard for sequence alignments, and can be used to store unaligned reads/sequences, but when used for unaligned sequences also contains unused alignment data that would bloat the file.

An alternative way to compress a DNA sequence file is via 2-bit (or 4-bit encoding) which can compress the sequence content by 75% (in 2-bit encoding, each nucleotide can be represented by 2-bits in the file eg A: 00, G: 01, C: 10, T: 11). I do not know of a generalizable tool that will compress/decompress aside from rolling your own. Further, the data must be de-compressed prior to streaming it into any alignment/sequence software. For general purpose compression algorithms, bzip2 can outperform (at least in compression) other algorithms (eg zip, gzip)

$\endgroup$
  • $\begingroup$ Hi, thx for the answer. I was thinking about the 2 bit compression myself but as you pointed out it needs to be decompressed for currents softwares. My 'problem" with BAM file is the as you said it contains many extra data that is useful for alignment but not otherwise. $\endgroup$ – Nandor Poka Mar 24 '15 at 23:02
  • $\begingroup$ With a 2-bit encoding you cannot have Ns, right? $\endgroup$ – cagliari2005 Mar 25 '15 at 6:18
  • 1
    $\begingroup$ You can, the format isn't exactly 2 bits per base. Instead it's actually 4, with the additional 2 bits being a mask that can convey information like, "it's an N", or "it's soft-masked". What this can't do is store quality information (or IUPAC ambiguity codes, as sometimes appear in fasta files). In fact, the space savings of a 2-bit variant format over straight fastq are 0.5 bytes per base in the sequence plus 3 bytes per read. Standard compression has better compression than that (you can, of course, compress 2-bit data). $\endgroup$ – Devon Ryan Mar 25 '15 at 8:05
1
$\begingroup$

BFQ = binary fastq Described here: http://ngsutils.org/modules/fastqutils/ I have not used it myself, but I gather that the maq aligner package can use (or generate) these files. Even though the bam format already incorporates the block compression (described by yourself and others), we routinely deal with gripped bam files and my sense is that there is some additional compression that takes place (I haven't calculated a percentage)

$\endgroup$
  • 2
    $\begingroup$ It should be noted that bfq can't store quality scores above 61 (it says it can go up to 62, but a T with a phred score of 62 is an N...). Anyway, a compressed bfq file is about 90% the size of a gzipped fastq file, so space savings are marginal (particularly given the added overhead of processing bfq). An uncompressed bfq file is about half the size of an uncompressed fastq file. $\endgroup$ – Devon Ryan Mar 26 '15 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.