Your answer to your titular question, How was the origin of CTVT parasitic cancer determined? is that cytogenic studies were done on the tumors. It arose in dogs and was logically assumed to be of dog origin, and was first characterized in 1876. The tumor cells were/are characterized by a rearranged karyotype, which is similar in tumors from in different parts of the world. (1969) Thus it was postulated that that the transmissible agent causing the CTVT is the tumor cell itself and that all worldwide CTVT have a clonal origin. (1985)
Your later question states:
What percentage / degree of genetic similarity is needed to identify something as belonging to a specific species?
When a tumor arises in an animal (say a dog), it is automatically assumed that the cancerous cells, regardless of their ploidy, arose from that animal (in this case, a dog), not a different animal. Not all animals have had their entire genome sequenced, so this question as stated isn't answerable (although the genome of the dog has been completely sequenced). And certainly, not all tumor cell lines have been completely sequenced. The assumption of origin is made by looking at certain genes (e.g. histocompatibility genes) and others known to be species specific.
Since there are relatively few transmissible cancers (that is, where the cancer itself acts as the infectious agent), this is still a safe assumption. While I'm fairly confident more transmissible cancers will be found, I'm also fairly confident that they will be species specific, because in order to escape detection, they will need to be recognized immunologically as "self", as they must survive in the host at least long enough to be transmitted.
This is the case with canine transmissible venereal tumor (CTVT). It is also the case with the contagious tumor Tasmanian Devil face tumor disease (DFTD). Just as with the canine infectious cancer, DFTD cells in different individuals have identical, complex chromosomal rearrangements and are genetically identical to each other but different to the host devil.
Spread of such an infectious cell line between unrelated individuals should be prevented by major histocompatibility complex (MHC) Class I antigens on the surface of the tumour cell, as Class I molecules play a key role in self/non-self-recognition. The recent emergence of the tumor (transmitted by biting) in the endangered marsupial Tasmanian devil (Sarcophilus harrisii) is an unfortunate exception. In the Tasmanian devil, the tumor is very aggressive and kills the host.
In the case of canine transmissible venereal tumor (CTVT), it was shown about five decades ago to be the same dog tumor being spread from dog to dog. The earliest studies were based on histological studies and karyotyping. Paraffin-embedded tumor samples still exist. More recently, with genetic analysis, it is possible to perform much more sophisticated testing on the tumor DNA.
The transmissible agent causing canine transmissible venereal tumor (CTVT) is thought to be the tumor cell itself. To test this hypothesis, we analyzed genetic markers including major histocompatibility (MHC) genes, microsatellites, and mitochondrial DNA (mtDNA) in naturally occurring tumors and matched blood samples. In each case, the tumor is genetically distinct from its host. Moreover, tumors collected from 40 dogs in 5 continents are derived from a single neoplastic clone that has diverged into two subclades. Phylogenetic analyses indicate that CTVT most likely originated from a wolf or an East Asian breed of dog between 200 and 2500 years ago. Although CTVT is highly aneuploid, it has a remarkably stable genotype.
likely originated at least 250 years ago from a wolf or an "old" Asian dog breed such as a husky or Shih Tzu...
Has been disputed. Estimates vary widely, from 11,000 years to 250 years.
Canine TVT: Evolution of Somatic Tumour Cell as Transmissible Parasites
MHC gene copy number variation in Tasmanian devils: implications for the spread of a contagious cancer
Clonal Origin and Evolution of a Transmissible Cancer