As far as I understand, heritability is defined as "proportion of variation of a phenotypical trait due to genetic variation between individuals in a population".
I see the concept is being applied on quantitative that can be measured with numbers, can be compared (i.e., $\mathrm{A}$ is taller than $\mathrm{B}$ by $x$ units of height) and varies continuously to form a distribution (e.g IQ forms a Gaussian distribution).
heritability in the narrow sense ($h^2$) has a mathematical definition. For a given quantitative trait $x$ of a population that forms a distribution with certain average and certain variance $V_P$ then $$h^2 = \dfrac{V_G}{V_P}$$ where $V_G$ is the variance in population due to variance in genetics and $V_P=V_G +V_E$ where $V_E$ is variance in population due to enviromental variation (I ignored other terms in the definition of $V_P$).
IQ has mean of 100 with $h^2=0.8$. So for a person whose IQ is 105 (5 points above average), genetic contribution contributed to this $0.8*5=4$ points difference and the remaining 1 point difference is environment.
I also see how the concept can be applied on qualitative traits (by this I mean traits that don't satisfy the properties of quantitative traits) even in the absence of well-defined notion of variance $V_P$ because the trait doesn't form a distribution. For example in a population of flowers that can only be red or blue.This red/blue trait does not have a well-defined $V_P$. We can still meaningfully say that this trait has $h^2=1$ since any variation in the color (red vs blue) is due to genetic variation. We can also speak meaningfully and say a certain qualitative trait has $h^2=0$ meaning any variation in the trait is due to different environments.
On the other hand, I think qualitative traits can have either $h^2=0$ or $1$ but no value in between, since $V_P$ is not well-defined (and hence $V_G$ and $V_E$ as well are not well-defined) for them.
Consider sexual orientation for example. Based on some twin studies it was estimated that homosexuality has $h^2=0.5$. But what does that mean really? how does homosexuality vary to form a variance? A person is either homosexual or not(heterosexual, bisexual or asexual).
My question is how to apply this $h^2=0.5$ value of homosexuality in the same way I applied it above on IQ?
More generally how to meaningfully be able to interpret heritiability of other qualitative traits (mental disorders for example like schizphrenia with $h^2=0.8$) that has has no well-defined $V_P$ and whose $h^2$ value is not $1$ or $0$ but rather lies between them?