Or as another example - what if you touch a surface that's contaminated with potentially pathogenic bacteria (like the ones at http://arstechnica.com/hardware/news/2008/05/study-keyboards-make-excellent-homes-for-nasty-bacteria.ars) and introduce those bacteria into your body and don't get sick. Then could there be some other damage that the bacteria could do your body? Like, at the very least, maybe promote a not-particularly-noticeable increase in inflammation in the regions where the immune system decides to fight the bacteria? Inflammation only starts to get noticeable when some really high threshold gets crossed.
By "sick", I mean, that the symptoms of the infection become noticeable enough to cross the threshold of being unpleasant to the person carrying the infection. I'm sure that there may be some sub-threshold effects that come in first.
And why is this important? We have to establish some sort of balance between using too much soap/other cleaning agents and not using enough of it. If we use too much, the bacteria could develop antibiotic resistance. And if we use too little, there could be an increase in the types of bacteria that cause chronic infections that don't get noticed, even though some of us might not even get physically sick more.