Yes, all female wasps are expected to have venom. In the case of social wasps, which I am inferring is what stung you, all female wasps are equally potent as they're morphologically the same. [This opens the question of how could you say this was a queen wasp.] The only difference between a wasp worker and a queen is that a worker is forced to fly around and forage by the queen so it doesn't have the time and energy to invest into laying eggs.
The following paper discusses this: http://dx.doi.org/10.1590/S1519-566X2008000600011
However, not all individuals are the same, both wasps and people, so you may have been stung by a rather weakened wasp/species to which you're not particularly sensitive.
I'd like to recommend the following paper which isn't about wasps, but discusses fundamental venom composition differences between fire ant workers and queens.
Fire Ant Queen Venom is Fire Ants' Secret Weapon
This is the DOI reference: http://dx.doi.org/10.1016/j.toxicon.2018.11.428
This is the DOI reference: https://doi.org/10.1101/454637
Contrary to wasps, most species of ants present a marked caste specialisation, meaning queens are quite different from workers. However, they still produce (and deliver) venom.
Thanks for your account!