Measure of diversity, accounting for known species

Question

I'm interested in measuring the diversity of each sample out of many independent samples. I know a priori all possible species that could appear in a sample, and expect the counts of species to be from 0 to about 10, although often times there will be none of a particular species within a sample. For example, some data might look like:

Sample # | Species A | Species B | Species C
--------------------------------------------------
1          0           4           0
2          1           1           1
3          0           1           1
4          2           2           2
5          0           1           0

The most diverse sample here is #4. The least is #5. Theoretically, #5 would also be the least diverse ever encountered (assuming there is at least one of some species in each sample).

I encountered trouble when using Simpson's Index of Diversity since that formula would make the diversities of Sample 2 and Sample 3 both 1.0. Intuitively, with this data, I am confident that #2 is more diverse. This is because I know all possible species that may show up.

I'm relying on this definition: $$ D = \frac{\sum n(n-1)}{N(N-1)} $$

Are there measures of diversity which account for a priori knowledge of possible species (and perhaps their bounds) and expect near-zero counts?

Answer 1

Simpson's estimator, which assumes that sample frequencies are not population frequencies, is unbiased on the probability scale. As @fileunderwater noted, it can be helpful to quantify diversity in terms of effective number of species, or Hill numbers. On this scale (the reciprocal of Simpson's concentration in this case), Simpson's estimator is no longer unbiased. Chao and Jost 2015 recently proposed some estimators for Hill diversity that formally rely on the total richness (of the sampled community) to estimate the true diversity of the community based on the sample, and argue that they have reduced the bias.

I sense, however, this is not exactly the same as your question. If I understand correctly, you have a known maximum diversity but not a known true cardinality/richness/total number of types within a given community/corpus whatever the pool you're looking at is called. In this case I might use maximum richness as a Bayesian prior; I might still use a Hill number as your metric but use the model to estimate that metric.

This 2014 review paper gets into these ideas a bit, or to take an Alan Turing perspective, you might find Zhang 2017 helpful.

Most diversity indices that ecologists use are not sensitive to total abundance, only relative abundance, so as Remi.b noted, classic indices (including Hill numbers) would all rank even communities with the same cardinality/richness identically; the only reason you're getting a difference between {1,1,1} and {2,2,2} with Simpson's estimator has to do with Simpson's solution to the bias in estimates of the community's diversity arising from using sample frequencies.