A professor of population genetics informed me that there is not a unique notion of pairwise Fst scores in the literature. This information has made me more cautious about what may be implemented by a given software package.
I am considering the GenAlEx Excel plugin (Peakall & Smouse 2005, Peakall & Smouse 2012) for computing pairwise Fst scores. The documentation states
Pairwise Fst : Outputs the pairwise Fst values between populations.
- Output Pairwise Matrix: Outputs pairwise population statistics as a triangular Matrix. Output is to worksheet
[NeiP]for Nei’s Genetic Distance, to[uNeiP]for Nei’s Unbiased Genetic Distance and to[FstP]for Pairwise Fst. This is in GenAlEx format for further analyses if required.- Output Labeled Pairwise Matrix: Outputs pairwise population statistics as a labeled triangular Matrix. Output is to worksheet
[NeiL]for Nei’s Genetic Distance, to[uNeiL]for Nei’s Unbiased Genetic Distance and to[FstL]for Pairwise Fst.- Output Pairwise Matrix as Table: Outputs pairwise population statistics as a table. Output is to worksheet
[NeiT]for Nei’s Genetic Distance, to[uNeiT]for Nei’s Unbiased Genetic Distance and to[FstT]for Pairwise Fst.
I gather from this that [NeiP] is for a function called Nei's genetic distance, but the Wikipedia page on genetic distance suggests that there are multiple distance functions named after Nei. This also confuses me because I thought that these functions named after Nei were distinct from fixation indices, and yet they are mentioned under this Pairwise Fst section in the documentation. I am similarly confused about the inclusion of [uNeiL] and [NeiT].
And then there is the mentioning of [FstP], [FstL], and [FstT]. Clearly the P, L, and T correspond to whether the function returns a matrix, labeled pairwise matrix, or the same but as a table.
I've probably misunderstood the documentation, but I didn't understand why Nei's distance was mentioned. But in any case, what is GenAlEx 6.5 actually calculating (i.e. formula/model) if I use any of [FstP], [FstL], and [FstT]?
I have emailed the authors of the GenAlEx6.5 package. I'll post an update if I get a response.
