Trying to understand the big picture behind DNA sequencing, alignment and searching

Question

I'm about to start a bioinformatics research project but I haven't any biological background.

I know my project is in regards to a performance analysis of DNA sequencing and searching "weapons" like Hadoop, Apache Spark and Apache Flink - so I've spent the last couple of days trying to put together the "DNA picture" before I get started with the programming stuff.

My understanding of the situation is that:

1. Next-generation sequencing (NGS) techniques are used to efficiently provide reads of DNA (conversions from real, physical DNA to something that can be read and analysed), however today's most practical methods provide reads that are short and in disorder.
2. Reads tell us which nucleotides, labeled one of ACGT, occur in sequence. Different nucleotides or values to take their place may exist, like N or X. Reads could range from 50 to 1000s of nucleotides in length depending on the method of sequencing.
3. You can find historical raw read data in various places online, including the Sequence Read Archive (SRA). The same website contains lots of other DNA/bio related information. Reads are commonly stored in .fasta files which follow a simple and practical standard. A single file could contain a very small or very large read or sequence.
4. Reads are then provided to DNA alignment programs like bowtie which will place them back in the correct order. The algorithms could use a template sequence to align them, or run "de novo" (without a template). The result of these alignments have also been indexed online however for the purpose of my studies I'll probably be aligning them myself.
5. Once aligned (or maybe during alignment), nucleotide differences from a template sequence can be programmatically found or searched for, with crossbow for instance. Note that many other tasks can too be performed - not only this search. If a particular substitution occurs in more than 1% of a what I think is called a "genome's" population, then it is called a Single-nucleotide Polymorphism (SNP or snip). Most SNPs have two alleles, or two different recorded nucleotides (like either G or C), but more than two is possible.
6. SNPs can be studied and mapped to various conditions or characteristics. A particular nucleotide could be responsible for part of one's emotional tendencies, reaction to particular medicines, or maybe anything, so a particular SNP could make a big difference.

What am I missing/what did I get wrong?

Answer 1

Here's a quick summary of a few mis-hits in your otherwise good analysis:

Not many bioinformatics applications use Hadoop, Apache Spark or Apache Flink. In fact, I have never heard of the Apache Spark and Flink tools, and I've seen only 2 people use Hadoop to process alignment files.

1. Reads are not "converted" real, physical DNA. They are representations of the signals from the molecules making up the DNA, as read by sequencing machines.
2. A,T,C,G are molecules making up DNA. N refers to "any of A,T,C,G", which translates to either unknown or ambiguous.
3. Reads are stored in FASTQ files. Sequences are stored in FASTA files. Reads include sequence as well as quality information, so, FASTA+QUAL=FASTQ
4. Re-ordering is a crude but approximate way to think about it. Remember, this process involves overlaps while reordering seldom does. Overlaps are crucial to the process of assembly/alignment. You are correct about the alignment to reference sequence and de novo assembly parts.
5. This is correct, though statistical models are applied to account for differences that are not necessarily significant variants (such as for sequencing errors)
6. Yes, variants, which include SNPs, can be correlated to differences in phenotype (traits). Emotional tendencies is something a bit too advanced, think of something more basic like diabetes or eye color.

The picture you have here is of a regular NGS analysis pipeline. This involves alignment/assembly, variant calling and biological analysis with relevant hypotheses. The assembly/alignment is the most computationally expensive part, and we use either HPC clusters or scalable cloud services such as AWS to get this done.

You should definitely talk to a biologist that has some computational experience to gain insight into the reason behind our analyses. Once you understand the motivations, your contribution will be more relevant and helpful to the community.

Answer 2

RAM's answer is very good, I'll just add on the computational side, short reads are error prone. That's important to account for when aligning or assembling. The reads themselves can just be inaccurate, which we detect by having multiple reads overlapping by a lot; we assume that stray discrepancies seen in only a single read at a position are errors. Also, if a reference genome is not quite close enough to the sample, reads might be misaligned.

The genomes of many organisms also have repetitive elements which can make it hard to correctly align reads, and hard to make an accurate reference genome.

And note that there can be many more discrepancies other than single nucleotide substitutions, though anything more involved can be challenging to detect with short read data alone. And most traits that are actually interesting to study are polygenic, so it's not exactly easy to say that one difference in DNA causes a detectable difference in phenotype.