Most of the things that come up in Google and Google Scholar for this question have a mostly/completely speculative tone, or at best use models instead of measurements. This report describes some actual measurements in pollen transfer to neighboring fields from GM fields, though they all seem to have been one-off measurements at some unknown time after the GM plants appeared, rather than investigations on if the genes are becoming more common over time, which might imply the ability to become unavoidably common in some environments or even a weed/pest/parasite. This doesn't seem like an obvious problem to me, because the traits associated with those genes are always meant for the context of being farmed or raised by humans and therefore probably wouldn't contribute more to fitness in wild populations relative to the wild alternatives of those genes, but a few counterexamples would smash that belief to bits.
Many organisms including plants, bacteria, insects, fishes and even mammals have been GMed. Most of them have experimental purposes, but GM crops have been released for commercial purposes. Maybe the most amazing between them is triticale, which is a hybrid of wheat (Triticum) and rye (Secale). Triticale, BT Potato, BT Tomato, BT Maize (I am not sure), BT and Golden Rice and lots of other GM crops have been released in almost all lands in world. Almost all these GM crops suppose to have higher yield and higher resistance to pathogens and physiological problems and as a result they have higher chance of surviving (specially in natural environment). A study published in Nature in 2001 reported that Bt-containing maize genes were found in maize in its center of origin, Oaxaca, Mexico. In 2002 Nature concluded that the evidence available is not sufficient to justify the publication of the original paper. A significant controversy happened over the paper and Nature's unprecedented notice. Also there are safety concerns about modified gene transfer during pollination in GM Rice(s). Even some scientists think that that this would not be a problem because rice is a self-pollinating plant. Any way GM crops can become abundant and release their genes by themselves or through gene transfer to other organisms.
I encountered this article discussing transgenic salmon, and it says that the modified genes don't find their way into nature because transgenic salmon are bad at the salmon mating ritual. It wasn't on purpose from what I can tell, but that's still a neat way to ensure modified organisms don't spread their genes into nature.
It's a relevant counterexample to what the question sought, but not a general argument against the possibility.