In the IUPAC Gold Book, IUPAC defines a polymer as follows:
A molecule of high relative molecular mass, the structure of which essentially comprises the multiple repetition of units derived, actually or conceptually, from molecules of low relative molecular mass
Fatty acids are relatively high in molecular mass compared to the monomer (which is essentially a carbon atom), although they're quite low in mass compared to all other biological polymers. Since we consider proteins polypeptides, as you rightly note, the monomers clearly don't have to be chemically identical, so the variations around a simple carbon that are present in various fatty acids should be fine.
The definition states nothing regarding typical numbers of repetitions or minimal numbers of repetition.
The IUPAC definition also mentions that for most (but not all) polymers, the specific number of repetition has little impact on the molecular properties of the polymer. Fatty acids do generally work more or less the same regardless of their specific length.
Sounds like fatty acids are carbon polymers to me, even if indeed we don't usually think of them like that.