As someone working with sequencing data and sequence files I was wondering whether is there a compression method / format for sequence / read files? Of course one can use general compressors like rar or zip to compress a fasta file, which is suitable for transferring data, but reading the sequence out of it is infeasible. Also I'm aware of BAM file format which is a block copressed gzip file of a SAM file, but as far as I know it is mainly used in alignments. But is there a compression for plain fastq(-like) files for storing millions of reads or a collection of sequences while being able to get sequence and quality info directly from the file?
BAM is the current standard for sequence alignments, and can be used to store unaligned reads/sequences, but when used for unaligned sequences also contains unused alignment data that would bloat the file.
An alternative way to compress a DNA sequence file is via 2-bit (or 4-bit encoding) which can compress the sequence content by 75% (in 2-bit encoding, each nucleotide can be represented by 2-bits in the file eg A: 00, G: 01, C: 10, T: 11). I do not know of a generalizable tool that will compress/decompress aside from rolling your own. Further, the data must be de-compressed prior to streaming it into any alignment/sequence software. For general purpose compression algorithms, bzip2 can outperform (at least in compression) other algorithms (eg zip, gzip)
BFQ = binary fastq Described here: http://ngsutils.org/modules/fastqutils/ I have not used it myself, but I gather that the maq aligner package can use (or generate) these files. Even though the bam format already incorporates the block compression (described by yourself and others), we routinely deal with gripped bam files and my sense is that there is some additional compression that takes place (I haven't calculated a percentage)