I think your question stems from your assumption that different human populations share an immune system that is fundamentally the same. This is not the case. In fact, some of the greatest genetic variability in human populations can be found in the human leukocyte antigen (HLA) gene complex. The HLA genes codes for the major histocompatibility complex (MHC) proteins in humans. These cell-surface proteins are involved in regulating the immune system.
Different human populations have different levels of innate immunity to certain pathogens based on their historical exposure to different pathogens at different frequencies over evolutionary time.
I'm going to use the generic terms 'indigenous person/people' to refer to non-European indigenous people who encountered Europeans for the first time between the 15th-late 18th centuries. They can be Aboriginal Australians, Native Americans/First Nations, Amerindians, etc.
The primary reason that indigenous people were historically so vulnerable to European diseases is that the Europeans had been exposed to these pathogens for a long time, which led to them developing resistance to the pathogens. This is natural selection at work. The individuals most likely to survive were those with an immune system able to fight off those particular diseases.
Prior to European contact, indigenous people would never have encountered the pathogens that the Europeans brought with them, they lacked any immunity to them as a result. This allowed contagious disease like smallpox to spread quickly and kill a lot of people these groups.
You might ask why didn't Native Americans' diseases wipe out the Europeans when they came into contact with them?
One of the main reasons for that was domesticated animals. Europeans, by mainly the luck of the draw, had lived in close contact for centuries with a large variety of domestic animals, and thus, got exposed to a large number of species-hopping diseases, thus developing immunity. Native Americans had very few domestic animals, and got far less exposure.
Riley E, Olerup O. HLA polymorphisms and evolution. Immunol Today. 1992
Sep;13(9):333-5. Review. PubMed PMID: 1361325.
Patterson KB, Runge T. Smallpox and the Native American. Am J Med Sci. 2002
Apr;323(4):216-22. Review. PubMed PMID: 12003378.
Markov PV, Pybus OG. Evolution and Diversity of the Human Leukocyte Antigen(HLA). Evolution, Medicine, and Public Health. 2015;2015(1):1. doi:10.1093/emph/eou033.
Guns, Germs, and Steel by Jared Diamond