The reason that an unequal sex ratio affects the effective population size is because offspring are produced by one male and one female parent, and an unequal sex ratio increases the rate at which genetic drift will occur.
"...the smaller number of males still contributes half of the genes in the next generation..."
In other words, assuming the male population is smaller, the genes passed on by the males/fathers are sampled from a smaller population than that of females/mothers. As you know, smaller populations are more prone drift.
Imagine the classic drift example with a bag of blue and red marbles, but have the parental generation with an unequal number of marbles in either bag.
Hartl & Clark illustrate (see figure 4 here) the relationship between sex-ratio and effective population size, where a sex ratio of 1:9 gives an effective population size of around 36% the actual size (given by $100 \times (N_e/N)$). This means that the rate of inbreeding and drift in the focal population is equivalent to that of an idealised population which is 0.36 times the census size of the focal population.
$$N_e = \frac{4N_mN_f}{N_m+N_f} = \frac{4 \times 1 \times 9}{1 + 9} = 3.6$$
The population is diploid and dioecious; each parent carries two genes, and each parent must be replaced within the population so each pair must produce two offspring. Consider the case where $N_m = N_f = 1$ (and thus $N = N_e = 2$). The number of genes in this population is four. Therefore, to maintain the same population size, four genes must be drawn from the ancestral population.
Further reading:
While (personally) I can't prove this mathematically, I can demonstrate it using simulation. I've written up a simulator of genetic drift in R, which can cope with unequal sex ratios.
The first four graphs are populations with $N_E = 360$ at various sex ratios ($N_M / N_F$ = 0.11, 0.33, 0.67, 1.00), for 100 replicate populations simulated over 80 generations, with an initial allele frequency of 0.2. The four simulations have different census population size, but equal effective population size, because of the sex ratio. The four groups of populations behave similarly (rates of drift are similar for the simulated populations under the various conditions).

These four use larger populations ($N \times 15$), at the same sex ratios, where $N_E = 5400$ in all cases simulated over 400 generations. Again the groups of simulations behave similarly.

Update:
I've added histograms to ease viewing of the pattern, using $P = 0.5$, for 1000 populations simulated over 25 generations, where $N_E = 360$ and census population size/sex ratio varies between simulations. This distributions are similar despite the different census population sizes.
