Background
The effective population size ($N_e$) is a central concept of evolutionary biology and is influenced by several parameters. For example: sex ratio bias affects $N_e$ $\left(N_e = \frac{4N_mN_f}{N_m+N_f}\right)$ and varying population size over time influences $N_e$ $\left(N_e = \frac{n}{\sum_{i=1}^n\frac{1}{N_i}}\right)$. There is a post on how overlapping generations influences population size.
Question
Selection also influences the effective population size. Intuitively, I'd expect that the higher is the fitness variance, the lower is the effective population size as fewer individuals contribute to the next generation. Am I right? How (what is the mathematical formulation) does fitness variance influences the effective population size?