From this Source: If generations are overlapping, then the effective population size $N_e$ does not equal the population size $N$.
I know mathematical formulations in order to find the effective population size $N_e$ when the sex-ratio is biased $\left(N_e = \frac{4N_mN_f}{N_m+N_f}\right)$ or when the population size varies cyclically through time $\left(N_e = \frac{n}{\sum_{i=1}^n\frac{1}{N_i}}\right)$.
What is the effective population size in a population with overlapping generations?
There are many different ways to do this, depending on what assumptions you make on e.g. stable age structure, distribution of offspring, haploidy/diploidy, population growth etc. As you probably know, there are also two main approaches to effective population sizes, namely ones based on:
An early attemp to calculate effective population size with overlapping generations comes from Felsenstein (1971), which is based on life table information. This paper includes several derivations, both on inbreeding $N_e$ and variance $N_e$. As an example, the formula of inbreeding $N_e$ for a diploid population is:
$N_e = \frac{N_1T}{1 + \sum_i^\inf l_i s_id_i v_{i+1}^2}$
Here, $T$ is the generation time, $l_i$ is the survival up to age i, $s_i$ is the survival from age i, $d_i$ is the probability of death at the end of age i (i.e. 1-$s_i$), and $v_{i+1}$ is the reproductive value of individuals in stage i+1. However, this is assuming a constant population size and a stable age distribution, so some rather restrictive assumptions. The paper also includes models for haploid populations, and I haven't gone through it carefully.
A couple of more recent papers that calculate $N_e$ for overlapping generations are Engen et al. (2005) and Engen et al. (2007). These papers use diffusion approximations to derive several formulas for $N_e$ under different assumptions, for age-structured density independent populations. One model for a haploid population in a fluctuating environment is:
$$\begin{align} N_e & = \frac{N}{\sigma_d^2T} \text{, with} \\ \sigma_d^2 & \approx \sum_{i=0}^k \frac{\lambda^{-2}}{u_i} \left[\left(\frac{\delta\lambda}{\delta b_i}\right)^2\sigma_i^2 + \left(\frac{\delta\lambda}{\delta s_i}\right)^2 s_i(1-s_i) + 2\left(\frac{\delta\lambda}{\delta b_i}\right)\left(\frac{\delta\lambda}{\delta s_i}\right) c_i \right] \end{align}$$
where $T$ is the generation time and $\sigma_d^2$ is the demographic variance. In the lower equation, $\lambda$ is the deterministic growth rate (dominant eigenvalue), $u_i$ is the proportion of the population in stage i (a component of the stable age distribution), $b_i$ is the expected number of offspring with variance $\sigma_i^2$, $s_i$ is the survival rate of stage i (with binomial variance), and $c_i$ is the covariance between reproduction and survival. You really need to dive into these papers to fully understand how these equations are derived and how the variables are defined (will take up far to much space here). However, at its core, it is using a diffusion approximation on an allele at a selectively neutral locus.
There are many other papers out there deriving effective population sizes, under different assumptions and restrictions, but the ones cited above should be a good starting point, and by looking at citations to and from these you should get a decent overview of the topic. Many textbooks also cover ways to calculate effective population sizes with overlapping generations, e.g. Hedrick (2011) and Felsenstein (2013, free pdf).
