This won't be a complete explanation: I'm bothered by these questions myself. But I'll say what I do know.
First:
Yes, Identity By Descent (IBD) is defined relative to a chosen threshold number of generations, at least in the sense that I understand it (in case there is more than one sense - I think there are ones allowing for mutation?). At the chosen number of generations back, we assume that all of those ancestors were unrelated. This seems troublesome! But two things might help it seem less so. First: when we ask if two individuals are related or not, we're really asking whether they're more closely related than the average relatedness of the background population. Second: the pedigree method of estimating kinship and inbreeding coefficients, is just that, a way of estimating, which may fail if its assumptions fail (e.g. if the ancestors at the threshold were not randomly chosen from a randomly mating population).
To illustrate, consider a question about a fictional individual with a fictional family tree (spoilers for Game of Thrones, season 1): How inbred is Joffrey Baratheon?
We know that Joffrey Baratheon is not, in fact, the son of King Robert Baratheon, but rather the product of secret incest between his mother Queen Cersei Lannister and her twin brother Jaime Lannister. If that were all we knew about the situation, we would draw the following pedigree:
By using this pedigree, we are implicitly assuming that Cersei and Jaime's parents are unrelated, i.e. randomly chosen from a large randomly mating background population (and assuming that if Cersei and Jaime are more likely than random individuals to share an allele, that this is due solely to IBD from one or the other of those randomly chosen parents, i.e. due to their being close relatives). Based on this assumption, applying the usual method to this pedigree, we estimate Joffrey's coefficient of inbreeding as $0.25$.
However, if we look deeper into the Lannister family tree, we realize that Cersei and Jaime are actually more closely related than a typical brother-sister pair: their parents were not randomly chosen, but were themselves related (though not closely enough to be scandalous for Westerosi society); their father Tywin Lannister, and their mother Joanna Lannister, were cousins. Using this information we draw a more complete pedigree:
Using the usual pedigree method again, we are now making a different assumption. We are no longer assuming that Joffrey's grandparents Tywin and Joanna were randomly chosen from the background population. We are instead assuming Joffrey's great-great-grandparents, Gerold Lannister and the Lady Rohanne Webber, were randomly chosen from the population. This assumption seems more reasonable. Since this new pedigree implies more inbreeding in Joffrey's ancestry, we guess that the method should give a higher estimate for his inbreeding coefficient: indeed it does, giving us an estimate of $0.28125$.
Here's how this illustrates the subjectivity of defining IBD. When we used the first pedigree, we were asking: What's the probability that both alleles at a locus in Joffrey's genome, are descended from the same allele in the grandparent generation? When we used the second pedigree, we were asking: What's the probability that both alleles at a locus in Joffrey's genome, are descended from the same allele in the great-great-grandparent generation? These are two different questions, and of course gave two different answers. The trouble, of course, is that the question of how inbred Joffrey is relative to the background population, only has one answer (which the pedigree answers may estimate more or less well)!
Second:
Although important coeffients like the "inbreeding coefficient" or "coefficient of relationship" are often presented in textbooks as being defined in terms of the probability of some pair of alleles being IBD, this cannot be the actual definition. This is because probabilities can't be negative, but relatedness can be (if two individuals are less related than average), as can the inbreeding coefficient (if the animal is outbred, i.e. its parents had negative relatedness). The possibility of negative relatedness even has interesting evolutionary implications. As a sort of flipside of Hamilton's rule for altruistic kin selection - where it can be evolutionarily beneficial for a gene to cause a bearer to harm oneself, in order to help a relative, i.e. an individual more likely than average to also carry a copy of that gene - negative relatedness makes it possible to have spiteful anti-kin selection - where it pays a gene to cause its bearer to harm itself, for no benefit at all, but solely to harm an individual less likely than average to carry a copy of itself (and more likely to carry its competitors)!
Indeed, in his original formulation of these and related coefficients, Sewall Wright did not invoke probabilities or identity by descent at all. For relatedness he talked about the correlation between individuals' allelic states (this requires assigning numbers to alleles, e.g. $A=1$ and $a=0$). Note that correlation coefficients can be negative. The "probability of IBD" interpretation was introduced by Malécot; it's made it into population genetics textbooks mostly just because it's easier to teach (despite the seemingly paradoxical subjectivity of reference number of generations, and the inconsistency with the possibility of negative relatedness).
Wright's explanation of the inbreeding coefficient is more intuitive: he points out that the most important effect of inbreeding is reduction in heterozygosity. Consider a randomly mating population: an individual randomly chosen from it, will have a certain fraction of its loci heterozygous. But an inbred individual will have fewer loci heterozygous (more loci homozygous). Malécot interpreted that excess homozygosity as being "due to coancestry [since a subjectively chosen reference generation]", but we can ignore that conceptual baggage, and just talk about the excess homozygosity (deficiency in heterozygosity) itself. Thus a better definition of the inbreeding coefficient is
$F = \frac{H_e - H}{H_e}$
where $H_e$ is the heterozygosity you would expect of the offspring of a random mating given the population's allele frequencies, and $H$ is the inbred individual's actual heterozygosity. A good presentation of this interpretation appears in Hartl's Primer of Population Genetics.
Note that this definition, unlike the probability definition (but like the correlation definition of relatedness) can be negative: outbred individuals have less homozygosity (more heterozygosity) than expected under random mating.
