I'm about to start a bioinformatics research project but I haven't any biological background.

I know my project is in regards to a performance analysis of DNA sequencing and searching "weapons" like Hadoop, Apache Spark and Apache Flink - so I've spent the last couple of days trying to put together the "DNA picture" before I get started with the programming stuff.

My understanding of the situation is that:

  1. Next-generation sequencing (NGS) techniques are used to efficiently provide reads of DNA (conversions from real, physical DNA to something that can be read and analysed), however today's most practical methods provide reads that are short and in disorder.
  2. Reads tell us which nucleotides, labeled one of ACGT, occur in sequence. Different nucleotides or values to take their place may exist, like N or X. Reads could range from 50 to 1000s of nucleotides in length depending on the method of sequencing.
  3. You can find historical raw read data in various places online, including the Sequence Read Archive (SRA). The same website contains lots of other DNA/bio related information. Reads are commonly stored in .fasta files which follow a simple and practical standard. A single file could contain a very small or very large read or sequence.
  4. Reads are then provided to DNA alignment programs like bowtie which will place them back in the correct order. The algorithms could use a template sequence to align them, or run "de novo" (without a template). The result of these alignments have also been indexed online however for the purpose of my studies I'll probably be aligning them myself.
  5. Once aligned (or maybe during alignment), nucleotide differences from a template sequence can be programmatically found or searched for, with crossbow for instance. Note that many other tasks can too be performed - not only this search. If a particular substitution occurs in more than 1% of a what I think is called a "genome's" population, then it is called a Single-nucleotide Polymorphism (SNP or snip). Most SNPs have two alleles, or two different recorded nucleotides (like either G or C), but more than two is possible.
  6. SNPs can be studied and mapped to various conditions or characteristics. A particular nucleotide could be responsible for part of one's emotional tendencies, reaction to particular medicines, or maybe anything, so a particular SNP could make a big difference.

What am I missing/what did I get wrong?

  • $\begingroup$ Did you find an exemplary/representative application that highlight usage of Spark or Flink (Hadoop MapReduce is dying AFAIK)? $\endgroup$
    – Phil
    May 25, 2016 at 18:15
  • $\begingroup$ @Phil It appears as though Flink has yet to gain significant traction, but Spark is well on its way. This is observable as a rough trend in table 1 of my draft literature review (bilalakil.github.io/mphil/assets/literature-review.pdf) - there you can also see that Hadoop isn't quite dying yet. Again, that's a very rough form of measurement, however it should be fairly indicative of trend. I've yet to describe such a explary/representative use of Spark thus far. $\endgroup$
    – Bilal Akil
    May 26, 2016 at 22:57

2 Answers 2


Here's a quick summary of a few mis-hits in your otherwise good analysis:

Not many bioinformatics applications use Hadoop, Apache Spark or Apache Flink. In fact, I have never heard of the Apache Spark and Flink tools, and I've seen only 2 people use Hadoop to process alignment files.

  1. Reads are not "converted" real, physical DNA. They are representations of the signals from the molecules making up the DNA, as read by sequencing machines.
  2. A,T,C,G are molecules making up DNA. N refers to "any of A,T,C,G", which translates to either unknown or ambiguous.
  3. Reads are stored in FASTQ files. Sequences are stored in FASTA files. Reads include sequence as well as quality information, so, FASTA+QUAL=FASTQ
  4. Re-ordering is a crude but approximate way to think about it. Remember, this process involves overlaps while reordering seldom does. Overlaps are crucial to the process of assembly/alignment. You are correct about the alignment to reference sequence and de novo assembly parts.
  5. This is correct, though statistical models are applied to account for differences that are not necessarily significant variants (such as for sequencing errors)
  6. Yes, variants, which include SNPs, can be correlated to differences in phenotype (traits). Emotional tendencies is something a bit too advanced, think of something more basic like diabetes or eye color.

The picture you have here is of a regular NGS analysis pipeline. This involves alignment/assembly, variant calling and biological analysis with relevant hypotheses. The assembly/alignment is the most computationally expensive part, and we use either HPC clusters or scalable cloud services such as AWS to get this done.

You should definitely talk to a biologist that has some computational experience to gain insight into the reason behind our analyses. Once you understand the motivations, your contribution will be more relevant and helpful to the community.

  • $\begingroup$ Mind my naivety, but if A, G, C and T are molecules, then what's a nucleotide? $\endgroup$
    – Bilal Akil
    May 7, 2015 at 14:23
  • 3
    $\begingroup$ @BilalAkil Nucleotides are a type of molecule. A, G, C, and T are both molecules and nucleotides $\endgroup$
    – C_Z_
    May 7, 2015 at 15:07
  • 1
    $\begingroup$ A nucleotide is a phosphate, pentose sugar, and a nitrogenous base in the most general sense. The A/C/G/T are derived from the type of nitrogenous base affixed to the sugar-phosphate. $\endgroup$
    – CKM
    May 7, 2015 at 16:54
  • 1
    $\begingroup$ @BilalAkil I wished to avoid as much biology as possible, hence the term "molecule". a nucleotide is a biological molecule with a sugar (ribose/deoxyribose) and a nitrogenous base (adenosine, guanosine, cytosine, thymidine) attached to it with a phosphate bond. The first part determines if the nucleotide is a part of DNA or RNA, and the second determines the base. DNA is a macromolecule, a super large molecule build from smaller constituent molecules :) $\endgroup$
    – Ram RS
    May 7, 2015 at 18:18

RAM's answer is very good, I'll just add on the computational side, short reads are error prone. That's important to account for when aligning or assembling. The reads themselves can just be inaccurate, which we detect by having multiple reads overlapping by a lot; we assume that stray discrepancies seen in only a single read at a position are errors. Also, if a reference genome is not quite close enough to the sample, reads might be misaligned.

The genomes of many organisms also have repetitive elements which can make it hard to correctly align reads, and hard to make an accurate reference genome.

And note that there can be many more discrepancies other than single nucleotide substitutions, though anything more involved can be challenging to detect with short read data alone. And most traits that are actually interesting to study are polygenic, so it's not exactly easy to say that one difference in DNA causes a detectable difference in phenotype.

  • $\begingroup$ Thank you. I did not wish to delve deep into the complexities we face. Pathogenic Single Nucleotide Variants in single-gene diseases are the easiest example of mutations with a phenotypical effect, aren't they? Note to OP: 1. Pathogenic, 2. single nucleotide, 3. single-gene 4. diseases, 5.phenotypical effect are the points of variability, each with at least 4 other entities that can take their place - the combinatorics give you the possible categories. Imagine the research when most of what we do it try and assign these categories to thousands of variants! $\endgroup$
    – Ram RS
    May 8, 2015 at 18:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .