Explaining the simpson's diversity index

Question

In my biology class, we were trying to compare the biodiversity of two site. In doing so, we were instructed to use the Simpson's Diversity index as follows:

$$ \frac{\sum_i n_i (n_i - 1)}{N (N - 1)}$$

Where $n_i$ is the total number of organisms of a particular species; and, $N$ is the total number of organisms of all species.

We were given a method to calculate the index in a class that required almost naught mathematical understanding for the purposes of the experiment. From what I have read, the Index calculates the probability that any 2 randomly selected organisms belong to the same species, and it is thought that the closer the result is to zero, the more diverse the region is.

What I can't understand, is the math behind the index; for example: why is it multiplying n by n-1? And how is this better than dividing the sum of all the organisms by the number of species?

I'm in Pre-Cal, so when explaining, please do so with minimum emphasis on "advanced" math terms.

Answer 1

That's an estimator for Simpson index, $$ \sum_i p_i^2 $$ where $p_i$ is the probability that $i$-th species is sampled. This is sum of the probability that two independent samples are from the same species. There are multiple ways of estimating this quantity from observations $n_i$.

When $n_i$ is small, $$\hat{p} = \frac{n_i}{N}$$ is unbiased but has high variance, and plugging it in $\hat{p}$ to the definition would also have high variance. (Note that this is equivalent to sampling with replacement.)

If one uses sampling without replacement formula, $$\hat{p^2} = \frac{n_i}{N} \cdot \frac{n_i-1}{N-1}$$ then, the resulting estimator is $$ \sum_i \frac{n_i \cdot (n_i - 1)}{N \cdot (N - 1)}$$

Note that both estimators converge to Simpson index for $N \to \infty$. But of course, there aren't infinite number of samples, usually.

Answer 2

For the following answer, assume we have two communities of size 4. Community 1 has two organisms of species A and two organisms of species B. Community 2 has three organisms of species A and one organism of species B.

The probability that two randomly selected organisms belong to the same species

In this example, for either community, the probability that two randomly selected organisms belong to the same species is the sum of the probability of selected two organisms of species A and two organisms of species B. In other words:

P(selecting two of the same species) = sum(P(selecting two of species X))

Where we sum over all species present (in this case, species A and species B)

The probability of selecting two organisms of species X is the probability that the first organism is species X times the probability that the second organism is species X.

The probability that the first organism is species X is simply:

P(first organism is species X) = nX/N

Where nX is the total number of organisms of species X and N is the total number of organisms in the community.

The probability that the second organism is of species X is:

P(second organism is species X) = (nX-1)/(N-1)

Because we are sampling without replacement, we subtract one from each value as we are now sampling from a community that is missing one individual (the one we first sampled).

The probability that two randomly selected organisms belong to species X, is then the sum of the previous two probabilities:

P(Selecting two of species X) = nX/N * (nX-1)/(N-1)

or

P(Selecting two of species X) = nX(nX-1)/N(N-1)

Then, to get the probability that two randomly selected organisms belong to the same species, we sum the above equation for all species. Summing over all species will give us your first equation.

In this case, for both communities we sum over species A and species B.

P(Selected two of same species) = [nA(nA-1)/N(N-1)] + [nB(nB-1)/N(N-1)]

P(Selected two of same species) = [nA(nA-1) + nB(nB-1)]/N(N-1)

For community 1, this comes out to 0.333. For community 2, it comes out to 0.5, accurately representing that community 1 is more diverse.

Why can't you take the total number of organisms and divide it by the total number of species?

The point of the Simpson's biodiversity index is not just to represent the total number of species in a community, but to portray how spread-out organisms are among the species; a community in which one species dominates and the rest are rare is considered less diverse than a community with the same number of species that has a roughly even number of individuals per species.

In the case of our two communities, the total number of organisms divided by the total number of species will give us the same number (4/2), not accurately representing that the communities do not have the same diversity (they have a different spread of organisms)