Indeed, this is all due to the crossing over between different alleles. For example, put the case of one cell that contains 2 chromosomes, one from the mother and another from the father. Each chromosome contains two alleles: Ab (mother) and aB (father).
The first thing that they will do is divide into two chromatids each one getting to: Ab[1M] (one mother's chromatid) and Ab[2M] (second mother's chromatid) and the same for the father aB[1F] and aB[2F]. Notice that the chromatids still joined by the centromere.
Now, the crossing over comes: in this case, if the A[2M] from mother's chromosome gets exchange with a[1F] of the father's chromosome, we get: Ab[1M], ab[2M], AB[1F], aB[2F].
As you see, when they segregate you will get 4 different cells with two alleles: Ab, ab, AB and aB.
I hope this answers your question.