# Calculating pairwise Fst from allele frequencies

I am trying to refresh my memory on population genetics for a project, and I'm having some trouble.

Assume I have genetic data on 3 populations with 5 individuals in each of a given species, with the following number of genotypes:

      0101 0102 0202 0103 0203 0303 0204
Pop1   2    3    0    0    0    0    0
Pop2   1    1    1    1    0    1    0
Pop3   1    0    3    0    0    0    1


I have named alleles 01-03. So 0101 is a homozygote individual for allele 01, 0102 is a hetorozygote individual etc.

Is it possible to do this, from the following formulae:  If I input the information above into Genepop to calculate pairwise Fst using the following dataset:

Test Genepop
TestLocus
pop
Pop1, 0101
Pop1, 0101
Pop1, 0102
Pop1, 0102
Pop1, 0102
pop
Pop2, 0102
Pop2, 0103
Pop2, 0303
Pop2, 0101
Pop2, 0202
pop
Pop3, 0101
Pop3, 0202
Pop3, 0202
Pop3, 0202
Pop3, 0204


it produces the following result:

Pop    1       2
2    0.0278
3    0.2308   0.0669


However, how can this result be reproduced and calculated by hand?

# The original Wright's formulation

$$Fst=\frac{var(p)}{\bar p}$$

, where $$\bar p = \frac{1}{J} \sum_{j=1}^J p_{j}$$ and $$var(p) = \frac{1}{J}\sum_{j=0}^J (p-\bar p)^2$$ is the variance in frequencies among populations.

This original formulation is valid only for a bi-allelic locus. Nei (1973) generalizes this definition to loci that have more than two alleles.

# Nei (1973) general formulation

I recommend to have a look a the paper (Nei 1973), it is a really nice paper.

Nei shows that defining that within Subpopulation gene diversity (a.k.a. expected heterozygosity) $$H_S$$ and the Total gene diversity (a.k.a. expected heterozygosity) $$H_T$$ as

$$H_S = \frac{1}{J}\sum_{j=1}^J \frac{1}{I}\sum_{i=1}^I p_{ij}^2$$

$$H_T = \frac{1}{I}\sum_{i=1}^I \left(\frac{1}{J} \sum_{j=1}^J p_{ij}\right)^2$$

, where $$I$$ is the number of alleles and $$J$$ is the number of populations. Then Wright's definition of $$F_{ST}$$

$$F_{ST} = \frac{var(p)}{p(1-p)}$$

generalizes (for cases with more than one allele) into

$$Fst = \frac{H_T-H_S}{H_T}$$

Have a look at Nei 1973 to understand why they are equivalent.

# Weir and Cockerham (1984) formulation for several loci

Averaging $$Fst$$ over several loci is more complicated than one would expect at first place. The solution is given in Weir and Cockerham 1984.

The formulation is a little cumbersome so I just invite you to have a look at the paper.

• Thank you for your reply, Remi.b. I am aware of both papers, but I will read through them again. In this case, I was mostly interested in the practical approach of solving a calculation as the one above. Would you say, that it is not possible from the formulars I have listed? Nov 19 '15 at 9:39
• I am not sure I understand what you mean. You have two formulas, one defining $H$ and one defining $Fst$ from $H_T$ and $H_S$ (but not $H$). How would it be possible to calculate $Fst$ from the frequencies in each population $j$ ($p_j$)? Nov 19 '15 at 17:37