When you get sick, you generally don't contract enough bacteria at once for them to succeed in battling your immune system, right? Their numbers must gradually increase in the host's body before they know that they can attack. How does that work?
I think the current answer to this for bacterial infections is quorum sensing. Quorum sensing is a signalling pathway in bacteria which senses a molecule that the bacteria themselves secrete. When the concentration of the quorum signal reaches a certain level, the bacteria interpret this as their population density reaching some threshhold.
Bacteria are always around - even infectious Staph, as described in the other answer, the bacteria are always being cleared out by the immune system, but when they find the right place where they can get critical mass, they dig in, form a biofilm and secrete toxins, which can help them divide more successfully.
This is a description of the process from a paper on Staph infection, a common bacterial infection in humans.
Viruses are typically simpler - as @MattDMo describes. They seem to rely on finding an environment where they can infect more cells at such a rate greater than the immune system can clear them out. For influenza, this is more of a 'shock and awe' offense where they infect and multiply quickly, trying to outpace the immune system. Another virus like HIV will actually multiply so slowly that the immune system can't find the host cells, allowing HIV to slowly spread over the host cell population.
Here's a bonus topic... When they were looking a cholera in the 1990s they found that V cholerae bacteria actually take up phage with toxins in them that convert benign bacteria to virulent ones (that cause dysenteric cholera). They do this by conveying both the pili that attach to human cells and the toxin that causes the violent G.I. reaction of dysentery. I think this is not unusual for bacterial diseases. (the reference is Science 272, 1910-14 BTW).
This has lead to a suspicion that bacteria might be even adapting to infectious phenotypes through genomic changes. This study had sequenced the genome of C difficile from 486 patients and found that relatively few of them really were capable of patient to patient infection:
Infectious agents like bacteria, viruses, fungi, etc., don't know when to "attack" or produce pathogenic substances, they just do it under their preferred conditions, and your body's immune system either succeeds in fighting them off immediately, or it doesn't and you get sick. Your body is constantly confronting and clearing potentially dangerous microorganisms without your ever being aware of it. Every breath of air, mouthful of food, or contact with your mucus membranes is a possible infection, if not for your innate immune system.
However, sometimes pathogens get past the first line of defense, or are otherwise able to evade it, and in the time it takes for the adaptive immune system to engage they are building up their numbers and gradually getting to the point where you realize that you are sick or otherwise infected.
Interestingly, many of the symptoms of sickness you often feel - being tired, feeling sore, having a fever or localized redness/inflammation - are actually caused by some of the effector chemicals or cytokines released by your own immune system, and not by the infective agents themselves.
For example, lipopolysaccharide or LPS is a major component of the cell wall of Gram-negative bacteria, including such pathogenic species as E. coli (food poisoning, urinary tract infections), Neisseria species (bacterial meningitis, gonorrhea), H. pylori (stomach ulcers), and Salmonella species (food poisoning, typhoid fever). LPS on its own does not cause fever or inflammation, for example, but when bound by certain immune cells (B-cells and macrophages, mostly) it rapidly and overwhelmingly promotes the generation and release of pro-inflammatory cytokines like TNFα and IL-1β. These proteins then act on target tissues to allow the inflammatory response - swelling, pain, increased temperature, redness, etc. - and recruit immune cells to fight whatever invaders must be there.