TO put simply: I am just wondering what mechanism or substance only enables small molecules to pass through the cell membrane. Is it cholesterol? My textbooks are not very clear.
The plasma membrane consists of a phospholipid bilayer which separates the internal environment of the cell from the external environment. It regulates the movement of substances in and out of the cell and helps keep a constantly regulated internal environment of the cell.
The plasma membrane consists of a phospholipid bilayer, which, as the name suggests, is made up of phospholipids(lipids that have a phosphate group attached to them.). The phospholipid has two regions a hydrophilic (water-loving / has an affinity for polar molecules) and hydrophobic (water-hating / has an affinity for nonpolar molecules).
So what happens is these molecules come together and arrange themselves in such a way that it faces the least amount of repulsions. Hence, the fatty acid tails (Hydrophobic part) face inside and the polar part(Hydrophilic part) face the outside, to look something like this.
Now, what happens is that when a molecule has to enter the cell it first has to interact with the polar part of the lipid membrane and then with the inner non-polar part of the membrane to pass through, hence if it is a polar molecule it will face repulsions and will not be able to pass through without the assistance of transmembrane proteins (it doesn't matter whether it is small or large, it simply won't), but, in the case of non-polar molecules, there is no repulsion from the fatty acid part of the phospholipids and hence it can pass through the membrane. When the non-polar molecules are too big some other mechanisms may be utilized (like receptor-mediated endocytosis).
With regard to your doubt regarding the "mechanism" with which a molecule passes through the membrane, in the simplest case, it will simply slip in between the phospholipids and through the bilayer. But, this is, as I said the simplest case, there may be transmembrane proteins which create channels for diffusion (aquaporin is a good example), or as I mentioned earlier they may be taken into the cell by endocytosis, through pumps like the Na-K pump etc.
And also cholesterol acts like a buffer to ensure that the spacing between the individual phospholipids is maintained so that it stays fluid (i.e. to maintain optimum rigidity of the membrane).
This is what an actual membrane looks like in an actual cell:
Hope you got a general idea.
You can refer to Chapter 11 of Molecular Biology of the Cell (by Bruce Alberts et al) if you want a more rigorous explanation.