When you get sick, you generally don't contract enough bacteria at once for them to succeed in battling your immune system, right? Their numbers must gradually increase in the host's body before they know that they can attack. How does that work?
I think the current answer to this for bacterial infections is quorum sensing. Quorum sensing is a signalling pathway in bacteria which senses a molecule that the bacteria themselves secrete. When the concentration of the quorum signal reaches a certain level, the bacteria interpret this as their population density reaching some threshhold.
Bacteria are always around - even infectious Staph, as described in the other answer, the bacteria are always being cleared out by the immune system, but when they find the right place where they can get critical mass, they dig in, form a biofilm and secrete toxins, which can help them divide more successfully.
This is a description of the process from a paper on Staph infection, a common bacterial infection in humans.
S. epidermidis is considered part of the of the normal human microbial flora, while S. aureus is usually regarded as a transient member. Colonization by either species usually does not lead to adverse events. However, when these organisms or their extracellular products are allowed to breach the epithelial layer, serious disease can result (1). S. aureus has many cell surface virulence factors (such as protein A and clumping factor) and secreted exotoxins and enzymes that allow strains to cause a myriad of infections. These diseases range from relatively benign furuncles and subcutaneous abscesses to scalded skin syndrome, sepsis, necrotizing pneumonia, and toxic shock syndrome (TSS). While no single cell surface virulence factor has been shown to be uniquely required for mucous membrane attachment, once colonization occurs, numerous secreted exotoxins, including the pyrogenic toxin superantigens and exfoliative toxins, definitively cause serious human disease. Other secreted exotoxins, such as the four hemolysins (α, β, δ, and γ) and Panton-Valentine leukocidin have also been suggested to contribute to significant illnesses. S. epidermidis does not possess the array of extracellular toxins that S. aureus does, and its primary virulence factor is considered to be its ability to form biofilms.
To improve their ability to cause this variety of human disease and to occupy numerous niches within the host, staphylococci have developed quorum-sensing systems that enable cell-to-cell communication and regulation of numerous colonization and virulence factors.
Viruses are typically simpler - as @MattDMo describes. They seem to rely on finding an environment where they can infect more cells at such a rate greater than the immune system can clear them out. For influenza, this is more of a 'shock and awe' offense where they infect and multiply quickly, trying to outpace the immune system. Another virus like HIV will actually multiply so slowly that the immune system can't find the host cells, allowing HIV to slowly spread over the host cell population.
Here's a bonus topic... When they were looking a cholera in the 1990s they found that V cholerae bacteria actually take up phage with toxins in them that convert benign bacteria to virulent ones (that cause dysenteric cholera). They do this by conveying both the pili that attach to human cells and the toxin that causes the violent G.I. reaction of dysentery. I think this is not unusual for bacterial diseases. (the reference is Science 272, 1910-14 BTW).
This has lead to a suspicion that bacteria might be even adapting to infectious phenotypes through genomic changes. This study had sequenced the genome of C difficile from 486 patients and found that relatively few of them really were capable of patient to patient infection:
The results of the study indicated that, although transmission between patients is likely to occur, it actually happens at relatively low frequency. In particular, concerns that healthcare teams were spreading infection between different hospitals seem to be misplaced. One exception to this general finding is that there were a large number of cases of infection from one particular strain that does appear to have been due to patient-to-patient transmission, emphasising the epidemic nature of this lineage. Notably, this strain has declined in UK hospitals in the last five years.
Infectious agents like bacteria, viruses, fungi, etc., don't know when to "attack" or produce pathogenic substances, they just do it under their preferred conditions, and your body's immune system either succeeds in fighting them off immediately, or it doesn't and you get sick. Your body is constantly confronting and clearing potentially dangerous microorganisms without your ever being aware of it. Every breath of air, mouthful of food, or contact with your mucus membranes is a possible infection, if not for your innate immune system.
However, sometimes pathogens get past the first line of defense, or are otherwise able to evade it, and in the time it takes for the adaptive immune system to engage they are building up their numbers and gradually getting to the point where you realize that you are sick or otherwise infected.
Interestingly, many of the symptoms of sickness you often feel - being tired, feeling sore, having a fever or localized redness/inflammation - are actually caused by some of the effector chemicals or cytokines released by your own immune system, and not by the infective agents themselves.
For example, lipopolysaccharide or LPS is a major component of the cell wall of Gram-negative bacteria, including such pathogenic species as E. coli (food poisoning, urinary tract infections), Neisseria species (bacterial meningitis, gonorrhea), H. pylori (stomach ulcers), and Salmonella species (food poisoning, typhoid fever). LPS on its own does not cause fever or inflammation, for example, but when bound by certain immune cells (B-cells and macrophages, mostly) it rapidly and overwhelmingly promotes the generation and release of pro-inflammatory cytokines like TNFα and IL-1β. These proteins then act on target tissues to allow the inflammatory response - swelling, pain, increased temperature, redness, etc. - and recruit immune cells to fight whatever invaders must be there.