How does it make sense to speak of heritability of qualitative traits?

Question

As far as I understand, heritability is defined as "proportion of variation of a phenotypical trait due to genetic variation between individuals in a population".

I see the concept is being applied on quantitative that can be measured with numbers, can be compared (i.e., $\mathrm{A}$ is taller than $\mathrm{B}$ by $x$ units of height) and varies continuously to form a distribution (e.g IQ forms a Gaussian distribution).

heritability in the narrow sense ($h^2$) has a mathematical definition. For a given quantitative trait $x$ of a population that forms a distribution with certain average and certain variance $V_P$ then $$h^2 = \dfrac{V_G}{V_P}$$ where $V_G$ is the variance in population due to variance in genetics and $V_P=V_G +V_E$ where $V_E$ is variance in population due to enviromental variation (I ignored other terms in the definition of $V_P$).

IQ has mean of 100 with $h^2=0.8$. So for a person whose IQ is 105 (5 points above average), genetic contribution contributed to this $0.8*5=4$ points difference and the remaining 1 point difference is environment.

I also see how the concept can be applied on qualitative traits (by this I mean traits that don't satisfy the properties of quantitative traits) even in the absence of well-defined notion of variance $V_P$ because the trait doesn't form a distribution. For example in a population of flowers that can only be red or blue.This red/blue trait does not have a well-defined $V_P$. We can still meaningfully say that this trait has $h^2=1$ since any variation in the color (red vs blue) is due to genetic variation. We can also speak meaningfully and say a certain qualitative trait has $h^2=0$ meaning any variation in the trait is due to different environments.

On the other hand, I think qualitative traits can have either $h^2=0$ or $1$ but no value in between, since $V_P$ is not well-defined (and hence $V_G$ and $V_E$ as well are not well-defined) for them.

Consider sexual orientation for example. Based on some twin studies it was estimated that homosexuality has $h^2=0.5$. But what does that mean really? how does homosexuality vary to form a variance? A person is either homosexual or not(heterosexual, bisexual or asexual).

My question is how to apply this $h^2=0.5$ value of homosexuality in the same way I applied it above on IQ?

More generally how to meaningfully be able to interpret heritiability of other qualitative traits (mental disorders for example like schizphrenia with $h^2=0.8$) that has has no well-defined $V_P$ and whose $h^2$ value is not $1$ or $0$ but rather lies between them?

Answer 1

The concept of heritability is a concept coming from quantitative genetics. It only applies to quantitative traits. This does not mean that the trait must be continuous. Discrete traits (such as the number of eyes for example) are quantitative traits.

For boolean traits, it is common to set one outcome to 0 and the other to 1. Which outcome is set to which value does not matter as it won't affect the variance. To take your example of homosexuality, one would classify all individuals as homosexual or heterosexual and set homosexual to 0 and heterosexual to 1 (or vice-versa) and compute the heritability from there. Of course, categorising people as either heterosexual or homosexual completely misses the diversity of sexual orientation. The ability to put its tongue in the vertical position would be a better example of a boolean trait.

As far as I know, for non-boolean nominal traits, the concept of heritability is undefined.

Answer 2

I think after some research I found the answer to the question. Measuring heritability of binary traits (e.g, diseases that do not conform to Mendelian rules like diabetes, mental disorders like schizophrenia and bipolar and etc) is based on the liability threshold model.

This theory assumes that every binary trait has underlying liability value that predisposes an individual for this trait. This liability varies continuously and forms a Gaussian distribution for a population. There's a threshold value of liability after which an individual becomes affected by this trait and it manifests itself (e.g, becomes schizophrenic). A person below the threshold value will be unaffected (e.g., does not have schizophrenia).

The theory posits that the source of variance $V_P$ of liability among the population is owing to genetic and environmental variances $V_G$ and $V_E$ respectively.

heritability as estimated by twin studies of monozygotic and dizygotic twins then estimates the heritability of the liability of a binary trait (e.g, homosexuality). In this case the formal definition of $h^2$ applies.