No. You are confused. There are different types of chromosomal sex determination. If one sex is heterogametic (an example of one system like this are mammals, which have an X and a Y chromosome) then the sexual phenotype (male versus female) is determined by the presence (or absence) of the Y chromosome. If you have a Y you will develop testes,and the androgens produced by the testes will direct all of your secondary sexual characteristics to develop the male fates.
In the absence of a Y chromosome the default developmental pathway of the somatic gonads is to form ovaries. The ovaries produce estrogen which will direct all of your secondary sexual
characteristics to develop female fates.
Gametes are either sperm or eggs. Early after fertilization some of the embryonic cells get sequestered away to form the primordial germ cells (PGCs). Later on these cells will mature in the somatic gonad to form the germline, giving rise to more sperm (or eggs). These germline cells have no effect on the sex or gender of a mammal. In some animals (like model organisms that can be subjected to genetic screens) you can isolate sexually transformed mutants where, for example the XX animals develop as males and produce sperm, and the XY animals develop as females and produce oocytes. There are other possible mutant situations as well where the male sex produces oocytes, and the female sex produces sperm.