I don't know if this is a proper answer... and this answer is really late... but if I remember correctly, the mitochondria pump H+ out to better compartmentalize the activities of the TCA and other oxygen-dependent metabolic processes within the mitochondria. This compartmentalization helps further minimize the possibility of a process or reaction to take place before the cell is ready. It also helps keep proteins that have closely related functions (as in they function in the same or complementary pathway) spatially close to each other.
Interestingly, prokaryotes have been observed to localize cellular activities as well. It is hypothesized that this occurs for the same or a similar reason as described above for eukaryotic cells, albeit it is likely less effective. This is accomplished through intracellular scaffolding of some kind of cytoskeleton. My guess is that prokaryotic cells can utilize this cytoskeleton to localize their proteins, but can't use it to localize H+ due to a lack of "binding" region (speculative).
Also, majority of prokaryotic organisms will pump the H+ into their periplasm (inter membranous space, not the "chaotic" external environment) as Rick Beeloo mentioned. This periplasm is a smaller environment than the cytoplasm and is more densely filled with proteins (and possibly others molecules) as well. I think the periplasm is more viscous as well - whether a result of solvent composition, high protein or other molecule concentration, or something else entirely, I'm not sure (relatively unfounded statement). Regardless, all these factors actually make it more difficult H+ molecules to diffuse away in the periplasm than in the cytoplasm.
Another argument against why the H+ must be pumped to the side opposite of the proteins is that ATP inhibits the TCA and other oxygen dependent metabolic processes. The ATP synthase creates ATP on the side that the H+ ion passively flows to. This causes feedback inhibition is much more temporally sensitive as ATP synthase synthesizes ATP on the same side as the proteins involved in ATP synthesis. I think that ATP synthase cannot produce ATP on the same side as the H+ ions because their is no energy to make the ATP with until the H+ ion has flowed down the concentration gradient across the membrane; by this I mean the energy used to create ATP can only be harnessed on the low-energy side of membrane (I'm guessing on this last sentence...).
As a result of these reasons and maybe more, I'm led to believe these are some reasons that it is advantageous to pump H+ ions out of the cell/mitochondria interior than in. Although 2 years late and by no means a definitive answer, I hope this adds some new perspectives or ideas to the discussion on why this process occurs as it does.