I'm interested in measuring the diversity of each sample out of many independent samples. I know a priori all possible species that could appear in a sample, and expect the counts of species to be from 0 to about 10, although often times there will be none of a particular species within a sample. For example, some data might look like:

Sample # | Species A | Species B | Species C
1          0           4           0
2          1           1           1
3          0           1           1
4          2           2           2
5          0           1           0

The most diverse sample here is #4. The least is #5. Theoretically, #5 would also be the least diverse ever encountered (assuming there is at least one of some species in each sample).

I encountered trouble when using Simpson's Index of Diversity since that formula would make the diversities of Sample 2 and Sample 3 both 1.0. Intuitively, with this data, I am confident that #2 is more diverse. This is because I know all possible species that may show up.

I'm relying on this definition: $$ D = \frac{\sum n(n-1)}{N(N-1)} $$

Are there measures of diversity which account for a priori knowledge of possible species (and perhaps their bounds) and expect near-zero counts?

  • $\begingroup$ Without genetic data, the standard other measures of biodiversity are Shannon-Wiener index and, simply, species richness (aka. number of species). I'm afraid that both Shanon-Wiener and species richness would rank sample 2 and sample 4 equally. $\endgroup$
    – Remi.b
    Commented Jan 10, 2019 at 9:03
  • $\begingroup$ There isn't a single "true" measure of diversity, and different aspects of diversity can be taken into account (abundance, evenness, uniqueness etc). Different indices also weight these aspects differently. So you need to carefully define what you are after before choosing an index. $\endgroup$ Commented Jan 23, 2019 at 13:00
  • $\begingroup$ Also, I question your calculation of D; to me the value for 2 & 4 would be 0.333 and for nr3 0.5 (i.e. 1-D equal to 0.667 and 0.5, respectively). $\endgroup$ Commented Jan 23, 2019 at 13:04
  • $\begingroup$ @fileunderwater Thanks for the comment. I realize there isn't a single "right" solution, I'm just looking for the most suitable: something that takes into account the number of unique species in addition to the overall abundance of each (perhaps with knowledge of possible species). Pardon me for not being from the biology community (this is actually a linguistics problem, but I'm reaching out to biology because there may be a fitting link) $\endgroup$
    – Alex L
    Commented Jan 23, 2019 at 19:49
  • $\begingroup$ @AlexL No worries about being from another field, especially since the diversity indices have quite wide appeal. I'm currently working a lot on scientometric data, and diversity indices (often borrowed from ecology) are common there as well. I think it would be useful for you to look at Hill-numbers, which basically are ways to rescale diversity indices (measures of entropy) into effective numbers of species. This makes it easier to compare numbers from communities/sites with differnt levels of species richness. $\endgroup$ Commented Jan 24, 2019 at 15:15

1 Answer 1


Simpson's estimator, which assumes that sample frequencies are not population frequencies, is unbiased on the probability scale. As @fileunderwater noted, it can be helpful to quantify diversity in terms of effective number of species, or Hill numbers. On this scale (the reciprocal of Simpson's concentration in this case), Simpson's estimator is no longer unbiased. Chao and Jost 2015 recently proposed some estimators for Hill diversity that formally rely on the total richness (of the sampled community) to estimate the true diversity of the community based on the sample, and argue that they have reduced the bias.

I sense, however, this is not exactly the same as your question. If I understand correctly, you have a known maximum diversity but not a known true cardinality/richness/total number of types within a given community/corpus whatever the pool you're looking at is called. In this case I might use maximum richness as a Bayesian prior; I might still use a Hill number as your metric but use the model to estimate that metric.

This 2014 review paper gets into these ideas a bit, or to take an Alan Turing perspective, you might find Zhang 2017 helpful.

Most diversity indices that ecologists use are not sensitive to total abundance, only relative abundance, so as Remi.b noted, classic indices (including Hill numbers) would all rank even communities with the same cardinality/richness identically; the only reason you're getting a difference between {1,1,1} and {2,2,2} with Simpson's estimator has to do with Simpson's solution to the bias in estimates of the community's diversity arising from using sample frequencies.

  • $\begingroup$ Thank you, this is good food for thought. The papers and book you referenced seem like valuable resources after a skim! $\endgroup$
    – Alex L
    Commented Feb 11, 2019 at 18:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .