34
$\begingroup$

I just had my whole genome sequenced and now I want to play a bit with bio-informatics. However, the documentation is quite scarce on the file standards used.

These are the files I have been given (the "raw" data):

Image of various file-format icons

Can someone explain me with some detail what is each file format, and for what are they used?

Transcribed list of file-formats from the picture:

  • Indel (TBI)
  • Indel (VCF)
  • SNP (VCF)
  • SNP (TBI)
  • CNV (TBI)
  • CNV (VCF)
  • BAM (BAI)
  • BAM
  • FASTQ R2
  • FASTQ R1
$\endgroup$
16
  • 8
    $\begingroup$ I'm voting to close this question as off-topic because isn't a biology question as defined for this site. Please take the tour and then go through the help pages starting with How to Ask questions effectively on this site for details. It might fit within Bioinformatics, but please check their help pages before posting there. Thanks! 😊 $\endgroup$
    – tyersome
    Apr 8, 2020 at 3:17
  • 13
    $\begingroup$ "I just sequenced my whole genome": Out of curiosity, how did you do that? $\endgroup$
    – bli
    Apr 8, 2020 at 7:48
  • 4
    $\begingroup$ Related: What is the difference between FASTA, FASTQ, and SAM file formats?. $\endgroup$ Apr 8, 2020 at 17:51
  • 4
    $\begingroup$ The machine they used was Illumina nextseq 6000, I paid ~250€, it takes about 2 months $\endgroup$
    – PedroD
    Apr 9, 2020 at 7:55
  • 5
    $\begingroup$ @PedroD Then perhaps the title should read "I had my genome sequenced..." $\endgroup$
    – J...
    Apr 9, 2020 at 17:40

1 Answer 1

61
$\begingroup$

This is a great biological question! It asks a lot about how empirical science is done in the field of modern biology. I'm glad we encourage such questions from curious people who want to learn more.


Your genome has a sequence that is unique to you, gifted to you by your parents and by small changes from random chance.

Your genome is pretty close to what is called the reference genome, which gets updated — infrequently1 — as sequencing platforms, pipelines, and algorithms change.

The reference genome is the "standard bearer". All other individual genomes are compared against it. This lets everyone compare apples to apples when trying to say something about the genome, in the aggregate or "big picture" view.

The files associated with the VCF format specify places in your genome that differ from the reference genome — and how they differ. These differences include:

  1. Indels — insertions or deletions of DNA, as compared with the reference.
  2. SNPs — single bases that differ from the reference.
  3. CNVs — repeated stretches of DNA that differ from repeats in the reference.

The specifics of each of these classes of variants can be associated with various diseases or other attributes that make you unique and interesting. More on that later.

Each of these VCF files in your dataset has an associated TBI file. This is an index file, which helps you to quickly retrieve elements of interest from these datasets using a command-line tool called tabix.

If you are looking for differences within a specific genomic interval — say, all the differences on chromosome 4, from bases 123456 to 567890 — then you can pass that interval to tabix along with your VCF file of interest, and it will return all the indels, SNPs, or CNVs within that range, if there are any within that interval. This subset is VCF-formatted, so by knowing the format, you can get more detail about those differences.

The BAM-formatted file collects all the individual sequencing reads in a form that is mapped or aligned to the reference genome, along with some summary information to tell you how those reads differ from the reference genome.

As with the VCF files, this BAM file has a BAI index file to go along with it. Like tabix, you can use the samtools command-line tool to extract the set of reads over your genomic region of interest, if you want to examine them further.

Both tabix and samtools are available from the same author. If you have questions about these tools, the Bioinformatics Stack Exchange site is a great place to ask questions3. Lots of bioinformaticists there have a great deal of experience with those toolkits and can help answer more specific questions.

The two FASTQ files are the raw sequence data coming off the sequencer, along with quality data. The sequence reads in these two files are paired for higher quality and longer reads, which makes the sequencing platform faster, cheaper, and better.

I walked a bit backwards, here, on purpose.

The raw FASTQ files are starting material for making the BAM file. The BAM file is then, in turn, used with variant calling software (example) to generate the variation (VCF) datasets.

Why walk backwards through these datasets?

Most people don't care about the raw FASTQ data, except perhaps to do manual alignment or check data quality. What is often more interesting or useful to most is how one's genome differs from the reference, which is captured in the variation data.

Those differences, in aggregate, are how companies like 23andme sell a product2 that tells you that you are more related to so-and-so population than so-and-so population, or whether you are more or less likely to have some disease or other odd, interesting phenotypic trait.

1. Even when new versions of this reference genome are released every few years or so, if you know what original reference genome was used to generate your BAM and VCF files, you can redo analyses based on that knowledge. There is likely some metadata associated with your sequencing results that you can look up to find this out, or you can ask the sequencing center from where you obtained these results.

2. For a bit more context, in this answer to another SE question, I talk a bit about SNPs and how they are used for testing association with so-called Mendelian disorders, diseases caused by single base changes, which may be of interest to you given what motivated you to get your genome sequenced, in the first place: https://biology.stackexchange.com/a/88839/5075

3. You can ask questions about them here, too, but you'll make certain people grumpy. Better to go to the other SE site for that.

$\endgroup$
5
  • 16
    $\begingroup$ OMG this is the most complete explanation I've read. Thank you! You should create a blog post with this, I could not find any that introduced all formats this way, actually the info is scattered around in an unorganized way. Thanks! $\endgroup$
    – PedroD
    Apr 8, 2020 at 9:34
  • 17
    $\begingroup$ +1, but it seems to be missing an essential piece of information. If the reference genome is regularly updated, how does one know which version of the reference genome the data is relative to? $\endgroup$
    – JBentley
    Apr 8, 2020 at 11:27
  • 2
    $\begingroup$ @JBentley you can't tell from the file formats (which this question is about). Sometimes the bam files can have indications, but the only way sure way is asking the person who generated the files. $\endgroup$
    – Nicolai
    Apr 8, 2020 at 13:06
  • 2
    $\begingroup$ @JBentley Reference genomes are not updated too frequently. But they do change every few years, and the selection of reference determines the outcome of BAM and VCF files, which are analysis products of the raw FASTQ reads. For humans, a couple common reference genome keywords are hg38 and, perhaps less so, hg19. Perhaps some metadata file not shown here in the question makes use of one of those labels, but as mentioned by @Nicolai, asking the sequencing center where the data come from should get an answer, as well. $\endgroup$ Apr 8, 2020 at 16:06
  • 3
    $\begingroup$ Please have an upvote! I'm a bioinformatician too, and this is an excellent answer. Also, just the fact that someone is able to ask this question suddenly brought into focus how far sequencing has moved on since I started. $\endgroup$
    – lupe
    Apr 9, 2020 at 15:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .