You are right that the membrane doesn't stop everything. Its main purpose is to separate the inside of the cell from the outside so that you can accumulate resources in it. Your potassium ions, proteins, sugars, RNA (and DNA, behind another membrane) have to stay inside the cell because they can't dissolve through it. There are indeed channels and transporters from some hydrophilic substances as you say, but they are usually very specific about what chemicals they will transport.*
The membrane is actually very delicate but it is reinforced by a cytoskeleton, and proteins are embedded in it to transmit force to features outside the membrane (such as desmosomes and hemidesmosomes connecting to other cells or extracellular matrix) so that the membrane itself is never put under strain. Think of it like a thin plastic bag that can hold water because you put it inside a much stronger container - even though the stronger container is not waterproof.
Many harmful chemicals would not be harmful if they could not get into your cells, so you should not be surprised if they are often hydrophobic and dissolve freely through the membrane. Some cells have other defense mechanisms (pumps to try to expel the toxins or enzymes to try to break them down), but ultimately any true toxin is defined as the one for which nothing works well enough.
Many pathogens have evolved methods of crossing the cell membrane. Bacteriophages famously break a hole in the membrane with a special injection apparatus, while many human viruses fuse a cell membrane section they carry around themselves with the victim cell membrane, using proteins that have evolved the right hydrophobic features. It is hard to defend against such things because our cellular membranes need to be able to fuse with themselves for purposes such as releasing vesicles of neurotransmitter. The same "no true pathogen..." argument applies - we only get sick from the pathogens that can infect us. If we lacked these protections, we would be even more vulnerable than we are.
- Exceptions, perforins and granzymes, prove the rule: they are meant to kill the cell affected.