The conservation biology literature has a great deal of information, particularly with reference to developing species survival plans (e.g., Traill et al.  report a minimum effective population size of ~4,000 will give a 99% persistence probability of 40 generations).
Because the question specifically mentions human populations, I'll focus my answer on the genetics of small human populations, though considerably less information is available.
Hamerton et al. (1965; Nature 206:1232-1234) studied chromosome abnormalities in 201 individuals from a total population size of 268 from the small island of Tristan da Cunha. These authors report increasing chromosome abnormalities (aneuploidy; hypo- or hyperdiploidy) with age and suggest that it may result in decreased mitotic efficiency. This population is thought to have developed from a founder population of only 15. According to Mantle and Pepys (2006; Clin Exp Allergy 4:161-170) approximately two or three of the original settlers were asthmatic, which has led to a very high prevalence (32%) in the current population.
Kaessmann et al. (2002; Am J Hum Genet 70:673-685) present a more modern study of linkage disequilibrium in two small human populations (Evenki and Saami; ~58,000 and ~60,000 population sizes, respectively) compared to two large populations (Finns and Swedes; ~5 and ~9 million). The authors find significant LD in 60% of the Evenki population and 48% of Saami, but only 29% in Finns and Swedes.
Lieberman et al. (2007; Nature 445:727-731) discuss the potential for human kin detection to avoid inbreeding. Such mechanisms have been found in other species, "from social amoebas, social insects and shrimp, to birds, aphids, plants, rodents and primates." Lieberman et al. propose mechanisms contributing to sibling detection in humans, including "maternal perinatal association" and "coresidence duration." Beyond these behavioral cues, the authors also suggest physiological cues such as major histocompatability complex as playing a role.