Are there any examples of proteins that, without coming from a recent duplication event, underwent a mutation(s) that caused it to have a novel interaction with a new ligand, substrate, other protein or molecule? By novel I imply that it doesn’t revert to an ancestral interaction or develop an interaction that is already performed by a close relative. I imagine that such mutations would be extremely rare, and that duplication is almost always necessary to have them fixed in a lineage.
The polymorphism in the gene product of the human ABO locus seems to provide an example of what the question demands. It does not involve a gene duplication or recovery from loss of function; rather it involves changes that cause interactions with a different ligand, and the mutations that produce these changes have been identified and examined at a molecular level.
I have described the biochemistry of the human blood group system in an answer I gave to a previous question, so the cartoons of the reactions are reproduced from that answer. The diagrams of the molecular interactions and the gene cluster are new.
Human AB polymorphism
The products of the A and B alleles of the human ABO locus are glycosyltransferases:
One of the enzymes recognizes the sugar, galactose (activated by UTP), whereas the other recognizes N-acetylgalactosamine. There is no doubt that they are the products of different alleles of the same gene, even though this is a member of a gene cluster (see below).
The structures of both these enzymes have been determined and the differences of the interaction with their substrates at the active site modelled as shown below:
There are four residues that differ critically between the two enzymes, three of which are shown. Two of these (Met/Leu 266 and Ala/Gly 268) make contact with the UDP-glucosyl substrates. The third (Ser/Gly 235) appears to affect the conformation of the H-antigen co-substrate.
How is a change in function possible without gene duplication?
The poster expresses the opinion (my rewording):
“I imagine that such mutations would be extremely rare, and that duplication is almost always necessary to have them fixed in a lineage.”
Why, then, is gene duplication not necessary in this case? The reason is that the enzymes involved are not indispensable: the product is an elaborated branched cell-surface polysaccharide, the function of which can be served by a range of variants. Indeed, the polysaccharide is functional without elaboration, as in the case of blood group ‘O’, where the enzyme is inactive. However, as discussed in my answer to the previous question, there may be a ‘hidden’ function that has driven the evolution of the polymorphism in humans. It has been suggested that the blood cell surface antigens were used by parasites such as the malarial parasite to enter the cells, and that variation provide resistance (albeit temporary, i.e. until the parasite evolved in response).
Footnote: Gene Clusters of Glucosylating enzymes
The question specifically asks for examples that do not come from a recent gene duplication. However I feel it important to mention the gene duplications that have produced such changes in the gene cluster to which the ABO locus belongs. This is to make it clear that the polymorphism is distinct from such examples.
Animals have the ability to vary the sugars on their cells by selecting from a large repertoire of genes with the potential to specify different glycosylating enzymes. An example is the GT6 gene family (glycosyltransferase family 6):
There are about seven members of this family, not all of which are present in every species. A paper by Turcot-Dubois et al. gives a detailed comparison of the genes in human, rat, toad and zebrafish. I just reproduce the gene map for the human cluster in which the four members of the family are indicated by solid triangles:
The enzymes specified by some of these gene have been characterized by X-ray crystallography, e.g. in this paper by Gastinel et al..