An easy way to visualize the mistake in your thought experiment is to consider a bottleneck event, when the ancestral population was very small, maybe just a few individuals. This would mean that the entire current population is descending from just a few individuals. Your thought experiment is assuming that the "pyramid" of your ancentors is expanding all the time, while the bottleneck event will essentially result in the opposite situation with an inversion of the pyramid, such as:
Your assumption: Bottleneck:
xx xx xx xx x
xx xx x x
x x xx xx
x xx xx xx xx
(present time)
The reason why the number of ancestors is smaller in the bottleneck event is naturally that the population is inbred (all individuals are related). In contrast, your thought experiment is assuming that all individuals forming the pyramid of your ancestors have completely unrelated ancestry. This is equivalent to assuming that the subpyramids for all ancestors never overlap.
Your assumption is unreasonable since human generations are overlapping, the fecundity distribution of individuals is skewed, inbreeding is common and reproduction between more distantly related individuals will happen by pure chance. We also know that humans have historically gone through several population bottlenecks, see e.g. Hawks et al. (2000) and Lui et al. (2006) for an overview. An realistic model of your ancestry is more likely to be (highly idealized and compacted):
x
x x
xx xx
xx xx xx xx
xx xx xx xx
xx xx
x x
x
(you)
The issue we are discussing here is actually known as pedigree collapse, where you will eventually find overlap between your ancestors, which will mean that the pedigree collapses instead of growing exponentially.
Beside historical bottlenecks, there are also modern examples of extreme human inbreeding. Even if this is not directly related to your thought experiment, you might for instance want to look the pedigree of the Habsburg royal family. Here is the genealogy of Charles II - remember, narrow loops in pedigrees are seldom a good idea:

Here you can for instance see that Charles' father is also the uncle of his own wife. You can also see that only 5 persons enter in the family tree from the outside through marriages, which means that Charles only had 7 completely independent ancestors (ignoring relationships further back, outside of this tree). Going back 3 generations (sort of - hard to count with overlap), it is also apparent that Charles had a total of 8 ancestors instead of the 14 expected under your model. Another modern example of extreme human inbreeding is the Fundamentalist Church of Jesus Christ of Latter-Day Saints, where marriages between close relatives have been common along with high levels of rare birth defects. Both these examples definately show, in an extreme way, that you cannot assume that all your ancestors are unrelated.