My textbook states that repetitive self-pollination in the plants cause inbreeding depression which causes infertility thus plants develop techniques called outbreeding devices to encourage outcrossing. My question is how does inbreeding depression cause infertility?
One of the main contributors to inbreeding depression are recessive lethal alleles. Suppose you have an individual who has the genotype Aa, where A is the dominant, non-lethal allele and a is the recessive lethal allele. This recessive lethal allele is likely very rare in the population (selection would decrease the frequency of a recessive lethal allele), so our Aa individual can breed with almost any individual without consequence because most other individuals are AA, so the offspring of this individual will be either Aa or AA.
However, if the Aa individual breeds with itself or with a close relative who has the Aa genotype, its offspring will have a chance of inheriting the two aa alleles, which is lethal, decreasing the fitness of the Aa individual that bred with itself (or a close relative). This decrease of fitness resulting from self-breeding will cause selective pressure to outcross instead of self.
It is important to note that the recessive allele doesn't need to be strictly lethal for this effect to happen - if the recessive allele causes infertility (for isntance, failure to produce gametes or something similar) the same effect will happen.
In short, inbreeding can cause detrimental recessive alleles to become paired, causing lethality or infertility.