Drugs like tetracyclines, macrolides and aminoglycosides bind to prokaryotic ribosomes. It is interesting that our body too having mitochondria, which have prokaryotic ribosomes, there is little(?) effect seen. It can not be ruled out that the effect may be lesser due to them being organelles. Azithromycin's effect on parasites like Toxoplasma and Plasmodium is due to its effect on the plastid that these Apicomplexans have (Castro-Filice et al. 2014).
There are two general points that should be appreciated in relation to this question:
Your statement that mitochondria “have prokaryotic ribosomes” is a misleading simplification. Although mitochondria and plastids are thought to be derived from eubacteria — and their ribosomes have some similarities in antibiotic sensitivity — the structures of their large rRNAs differ (as indeed do those of their small rRNAs and their ribosomal proteins). This is important because the large rRNA (the equivalent of 23S rRNA in E.coli) is the target of many antibiotics, including the macrolide antibiotics, of which azithromycin is a member.
Experiments in vitro have shown that human mitochondrial ribosomes are not sensitive to some of the macrolide antibiotics that inhibit eubacterial ribosomes, whereas those of some lower eukaryotes are.
Therefore it should not be particularly surprising that macrolide antibiotics do not affect human mitochondria, or that they do affect Toxiplasma etc., the ribosomes in the apicoplasts of which are sensitive to these antibiotics in vitro — presumably because of their rRNA structure in the specific target region.
An answer to the question in your title would seem to require an explanation of the molecular basis of these different sensitivities. It is thought (although not, I gather, definitely proved) that this is the identity of the base at the position equivalent to 2058 of E.coli 23S rRNA: those ribosomes that are sensitive to the antibiotics have the base A in that position, whereas those that are resistant have the base G.
The endosymbiotic theory states that eukaryotic mitochondria were once freely-living bacteria that somehow migrated into the cell and began a symbiotic relationship.
However, just because the mitochondria within human cells have prokaryotic origins, these are distant relationships that are exceedingly old. The ribosomes of current day bacteria (on which some classes of antibiotics target) are NOT the same as the ribosomes of the mitochondria. The structures and physical properties of the mitochondrial ribosomes, as well as many of the genes within the mitochondrial DNA are significantly different than their previous prokaryotic ancestors. Here is a link to a open access manuscript that discusses some of the differences in human, bacterial and mitochondrial ribosomes.