Why is the quality range of fastq format so broad?

Question

Referring to fastq format, it is clear that in fastq format, there are 94 quality value for a sequenced Nucleic Acid of a DNA sequence read and they are:

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

So, I have two questions:

1. Why is the quality value range so broad (94 levels)?

2. In a particular case of analyzing data, we saw, the frequency of I is so high, which is not a good quality. So, why would we accept such a poor quality in research?

Answer 1

Initially, there were several quality encodings that used to follow different ranges of ASCII characters to denote the quality of read. The range that you mention is a union of all those encoding formats. Nowadays, the most common encoding is Phred+33 (used by Illumina, Sanger, Ion Torrent and other popular sequencers) which uses these characters:

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHI

Character I denotes a phred score of 40. Aligners, when they read fasta files, by default assign a score of 40 (I) to all positions. This might be one why reason you are seeing a lot of I.

Anyways, if your fastq follows the phred+33 format then 40 (I) is a pretty good score which denotes a good quality read. Or you could be aligning fasta files. It would be a bad score if your encoding is phred+64 which I guess, is not the case.

ADDITION

The file that you linked is a fastq file. This sequencing experiment has been done on 454 GS FLX Titanium machine. 454 machines use a different way of calculating quality scores compared to the traditional basecalling phred scores. From the manual:

6.6 Phred-equivalent Base Quality Scores

Quality scores for individual called bases are determined by a method developed in collaboration with the Broad Institute (Genome Research,18(5): 763-70, 2008), whereby the methodology described by Ewing and Green (Genome Research, 8: 186-194, 1998) for the creation of quality scores as part of the Phred basecalling algorithm is applied to generating quality scores for 454 Sequencing reads. The quality scores computed for each called base are written to the CWF and SFF files (and optionally to a file paralleling the basecall FASTA file). Briefly, the method compares the properties of each base’s flowgram signals against properties that have been found to correlate with accurate and/or error-prone signal information, using training sets of read data. A multivariate analysis of those properties determines the sets of property values that best describe “bins” of basecalls, then assigns the training set accuracy rates of the basecalls in each bin as a quality score using the following scale:

$$Q = -10\ \text{log}_{10}(\text{error rate})$$

Usually, you'll have two files for 454 experiments — one that holds the sequence and the other that holds the quality at each position. These files can be combined to generate a fastq. This file is encoded in the phred+33 (also called Sanger/Illumina 1.9) format which I mentioned above. Phred+33, I guess, is also the standard encoding format adopted by NCBI/ENA/DDBJ and all new sequencing runs are encoded in this format. If you do not know the encoding you can run fastqc on your fastq file and it reports the type of encoding.

In your case, I would denote a score of 40 (ASCII value - 33) which means an error rate of 10^-4 i.e. the read is of good quality.

The below graph is obtained by a fastqc run on your file. The bars denote quality statistics at each position. Note that the mean quality drops towards the end of the read.

A fasta file does not indicate quality but when you map them to your target, the aligner (I know of Bowtie based ones) assigns a default score of 40 at each position, during the alignment (and in the output).

Answer 2

As mentioned by @WYSIWYG in his answer the quality scores in FASTQ file format are encoded in ASCII characters, and there has been several ways to encode this information. The wikipedia FASTQ page that you link in your question describes some (not sure if all) the different alternatives.

Now, why do we have this wide range of characters for the quality scores comes down to the question of what these characters really mean. What we really care is about the probability that a particular base has been assigned incorrectly. This is the base call error probability (p). We can define a quality score based on the error probability as:

q = -10 x log10(p)

So, if your base call is truly wrong (probability of error p = 1) then q = 0. If your base call is certainly correct (p = 0) then q -> Inf as p -> 0. Imagine your probability of error is p = 0.0001 (that is, most likely the base is correctly assigned). Then q = -10 * log10(0.0001) = 40. Therefore, a score of q = 40 means a probability of error in that base call of 1/10000. Quality scores are considered, as far as I know, between 0 and 40.

Now, another issue is how to include this information for each base in the FASTQ file. It would take too much storage space to include the score as a real value number (i.e. with decimals). A possibility instead is to round it- after all there is no much difference between a score of say 39.7 and 40- we care more about the order of magnitude. But then we will have to still store two digit integer numbers per base. What we can do is map the rounded score to the corresponding ASCII character + some offset. That is, as you can see in the FASTQ page the ASCII value for character I is 73. If you are using phred-33 encoding this actually means that character I = 73 corresponds to quality score q = 73 - 33 = 40, which means p = 0.0001. Voila. Character ! = 33 represents score 33 - 33 = 0 (p = 1). In a phred-64 representation a score q = 40 is encoded with character h = 104. Then q = 104 - 64 = 40.

Of course this is just a rough approximation about how the actual phred score is mapped. Phred is actually the name of a program that was developed to maps the quality scores to ASCII characters. The method in more detail is explained in the original publication (which I admit had only went through):

Ewing B, Green P. Base-calling of automated sequencer traces using phred. II. Error probabilities. Genome Res. 1998 Mar;8(3):186-94.