I would imagine that the answer to this question would be population control, especially since even if one sibling is homosexual this does not necessarily mean that the other siblings will be too.
Several years back, results of a twin study (Zietsch, et al., 2008; popular press in The Economist) suggested that genes associated with homosexuality make heterosexuals who carry the same genes more reproductively successful. This would explain the observation that, for example, sisters of homosexual males have more offspring.
We show that psychologically masculine females and feminine men are (a) more likely to be nonheterosexual but (b), when heterosexual, have more opposite-sex sexual partners.
So if more sexual partners in heterosexuals is correlated with more offspring, then the genes associated with homosexuality may lead to more offspring in heterosexuals that carry them. So these "homosexuality" genes could easily be maintained in the population by conferring a reproductive advantage to siblings.
The authors conclude:
Taken together, these results suggest that genes predisposing to homosexuality may confer a mating advantage in heterosexuals, which could help explain the evolution and maintenance of homosexuality in the population.
Obviously selection would appear to not favour being homosexual, in an evolutionary sense it represents somewhat of a decrease in fitness: Homosexuals fail to reproduce successfully due to the requirement of both male and female gametes and reproductive organs, therefore significantly fewer than the average heterosexual couple. Certainly I don't think it would have evolved as some kind of population control method - group selection theory is generally discarded in favour of a gene-centred theory (see Dawkins for popular science literature)
There is debate about whether someone can be biologically "preprogrammed" to be homosexual, this can occur genetically or epigenetically. Genetic models have used kin selection, overdominance, sexual antagonism in the past and are briefly discussed in the following article on epigenetics. Recent work has looked at possible models by which heritable homosexuality could arise by epigenetic markers and this was covered in a lot of mainstream media (1,2.. just google epigenetics homosexuality). This quote is from their abstract and explains one reason it is difficult to pin down heritable causes of homosexuality:
Pedigree and twin studies indicate that homosexuality has substantial heritability in both sexes, yet concordance between identical twins is low and molecular studies have failed to find associated DNA makers. This paradoxical pattern calls for an explanation.
By their readily testable model they have shown possible (& plausible) conditions under which the "epi-marks" which cause homosexuality could spread through populations.
Genetic studies of twins where one is homosexual have also revealed links between genes and a mating advantage. For example if in a pair of male identical twins one is homosexual, the other has a mating success greater than that of the average male. However, it could be that this link is found because of social reasons (perhaps someone who has grown up with a homosexual sibling may have a different mentality or lifestyle socially which could help them increase reproductive success).
Note: I have no problem with a person's sexual orientation & the rights/ethics of homosexuality is not the topic of this question & answer. This is purely an answer to the obvious evolutionary conundrum.
This is not a subject I know well, but I can point to several recent textbooks on homosexuality in animals, all with an evolutionary perspective. I remember that Bagemihl's book got good reviews when it was published. In general, I think it is important to acknowledge that homosexuality is common in many animal species and not just humans:
There is an interesting theory coming from a slightly different, yet related, field. It was developed by Paul Vasey and Doug VanderLaan of the University of Lethbridge, they are both evolutionary psychologists:
- An Adaptive Cognitive Dissociation Between Willingness to Help Kin and Nonkin in Samoan Fa’afafine (or direct link)
They called their theory the "helper in the nest" hypothesis. It basically states that gay people compensate for their lack of children (this has been been covered thoroughly in the other answers) by offering an indirect benefit through enhancing the survival prospects of close relatives. And they do so by being “helpers in the nest”, by acting altruistically toward nieces and nephews.
Their study was conducted in Samoa, and they chose the country because males who prefer men as sexual partners are widely recognized and accepted there as a distinct gender category (called fa’afafine - neither man nor woman), which makes for a clearly defined sample for study. It's worth mentioning though that the authors see Samoan culture as very different from most Western cultures. Yet, they affirm Samoa’s communitarian culture may be more—not less—representative of the environment in which male same-sex sexuality evolved eons ago.
There are a lot of theories about how homosexual behavior might be maintained in a population or provide some evolutionary advantage. One thing to remember though is that there is a big difference between homosexual behavior and obligate homosexuality. One could quite easily confer benefits, like any other intra-sex social interaction. The other carries a huge fitness cost. However it's not clear that obligate homosexuality has been common in humans. It is in our current culture, but historically, it's possible that most people (particularly women) would have had children because of cultural pressure to do so, regardless of sexual orientation. Without knowing more about cultural influences on reproductive rates, we can't say if there even is a mystery to be solved here.
Humans have a wide variety of triggers for sexual arousal. I think that having a variety of triggers can beat having too few - ie it can result in more consistent reproductive success, especially within humans who do not have strong and obvious fertile periods and will have relatively few offspring. Where a strong sex urge exists and fertile mates are not available having other outlets - masturbation and homosexuality - may reduce the occurrences of harassment and rape and prevent strong sex drives being injurious or socially destructive.
Being homosexual does not prevent the ability to have hetero sex and many people who are homosexual by preference still persist in hetero relationships and have children. It isn't the preference but the inability to have hetero sex with fertile partners that would select against homosexual behaviours.
A recent study using a massive pool of genetic data suggests that indeed genetic factors associated with homosexuality also confer a mating advantage to heterosexual individuals in the sense of being "more attractive". Some quotes from the article:
According to an abstract the team submitted to the American Society for Human Genetics, whose annual meeting is under way this week in San Diego, the DNA signals linked to gay sexual experiences also appeared more often in straight men who had a larger number of sex partners. The team also notes that straight men with the gay-linked variants were, on average, judged more “physically attractive” than others (the researchers decline to say who did the judging). This, the scientists conclude, could mean that these variants also “confer a mating advantage to heterosexual carriers.”
Such trade-offs are a fact of evolution. For instance, gene variants that can cause sickle-cell anemia also lend protection against malaria. The resulting balance means the sickle-cell gene doesn’t die out. The researchers say their new findings about non-heterosexual behavior, though not conclusive, are consistent with such a Darwinian balancing act.
PS: I know this is in line with the most upvoted answer, but the nature of this study, I think, singles out the need for a new and more recent answer.