Well, your question as a whole is quite broad, with a lot of sub-questions or items (and some inaccuracies, as in the case of flatworms) . That being said, in this answer I'll address only your main question (and nothing more), which is ipsis litteris:
Is there a theoretical maximum on the number of sexes a given species of organism can have?
All the content in this answer is based on this excellent book from Michael Majerus, Sex Wars: Genes, Bacteria, and Biased Sex Ratios:
Unfortunately, I don't have my library anymore and I can't find this book online. So, instead of (boringly) quoting the author, I'll try to summarise his hypothesis. Of course, to a better explanation, you can read the book itself.
Let's start supposing that you're pretty comfortable with the concept of sexual reproduction and sexual types, which I think you are.
Suppose we have two sexual types, called A and B. They have receptors in their gametes' membranes, in such a way that you cannot fusion A + A or B + B gametes, but you can fusion A + B gametes. If the A gamete could fertilize another A gamete, you'll have all gametes in a given gonad fertilising each other, which makes no sense. So, will assume that a given sexual type has mechanisms to avoid fertilising the same sexual type.
So far, what I described is the common system with two sexes, be them "+ and -" or "male and female", it doesn't matter.
Now, let's suppose three sexual types: A, B and C. For this to work, their membrane receptors must be such that A can recognise B and C, but not A. This is theoretically possible (although it is a mess genetically speaking). Thus, the possible matings are:
- Sexual type A → A + B and A + C;
- Sexual type B → B + C and B + A;
- Sexual type C → C + A and C + B;
A system with 3 or more sexes has obvious advantages. Right now, for instance, in a population with two sexual types and the same sex ratio, one has 50% (1/2) of the individuals in that population as potential mate. But, supposing a sex system with 5 sex types (A, B, C, D and E), any individual would have 80% (4/5) of the individuals in that population as potential mate.
So, why cannot we find sexual systems with more than two sex types?
The answer is: we can, but not among species which perform fertilization (animals, plants, most of fungi, most of algae etc...).
This is the problem: in fertilization, or syngamy, the gametes don't simply join their nuclei. It's more complex than that: there is cytoplasmic inheritance, or extranuclear inheritance.
To keep the example among animals, the group you said you know best: mitochondria in the zygote comes from only one gamete (the female one). That happens to avoid what we call heteroplasmy, which is the presence of more than one type of organellar (mitochondrial, in this case) genome in the zygote.
In sex systems with only two sexual types the rule is very clear: "mitochondria comes from this one, but not from that one". That's it, case solved.
But what happens with our system with three (A, B and C) sexual types? If you define that when A mates with B only A donates the mitochondria and, similarly, if you define that when A mates with C only A donates the mitochondria, who donates the mitochondria when C mates with B? Do you see? There is no possible rule here!
Therefore, cytoplasmic inheritance is, probably, what avoids the evolution of more than 2 sexual types.
How can we support this claim?
Let's find what happens in species that perform sexual reproduction, but no fertilization (and, therefore, don't have the cytoplasmic inheritance problem). The best example are protozoans that perform conjugation.
Tetrahymena is a well known example. In these protozoans, according to Umen, 2013:
An individual Tetrahymena cell expresses one of seven possible mating types (I–VII) that it faithfully and indefinitely maintains during vegetative propagation. After mating each new exconjugant has a recombinant micronucleus formed from its two parents. However, rather than just expressing one of two parental mating types, the sexual progeny of Tetrahymena undergo a kind of genetic roulette that allows them to choose at random up to seven possible mating types. (emphasis mine)
So, we have sex mating types. How that system works? According to Cervantes et al., 2015:
The unicellular eukaryote Tetrahymena thermophila has seven mating types. Cells can mate only when they recognize cells of a different mating type as non-self.
Thus, if you belong to sexual type I you can mate with someone that belongs to sexual types II, III, IV, V, VI or VII. The same way, if you belong to sexual type II, you can mate with someone that belongs to sexual types I, III, IV, V, VI or VII, and so on...
The record, however, probably belongs to the basidiomycete Schizophyllum commune, with more than 28.000 sexual types! (Whitfield, 2004) To support the idea that cytoplasmic inheritance is in fact the main factor that avoids more than 2 sexual types, it's worth noting that biparental mitochondrial transmission is quite common in fungi (Barr, Neiman and Taylor, 2005).
Sources:
- Majerus, M. (2003). Sex wars. 1st ed. Princeton: Princeton university press.
- Umen, J. (2013). Genetics: Swinging Ciliates’ Seven Sexes. Current Biology, 23(11), pp.R475-R477.
- Cervantes, M., Hamilton, E., Xiong, J., Lawson, M., Yuan, D., Hadjithomas, M., Miao, W. and Orias, E. (2015). Correction: Selecting One of Several Mating Types through Gene Segment Joining and Deletion in Tetrahymena thermophila. PLOS Biology, 13(10), p.e1002284.
- Whitfield, J. (2004). Everything You Always Wanted to Know about Sexes. PLoS Biology, 2(6), p.e183
- Barr, C., Neiman, M. and Taylor, D. (2005). Inheritance and recombination of mitochondrial genomes in plants, fungi and animals. New Phytologist, 168(1), pp.39-50.
