You are correct in saying that whole-cell potency and in vitro/cell-free potency often differ by quite a bit. There are actually a lot of factors that can contribute to this, but I'll try to give a couple illustrative examples, for increased and decreased potency (when moving from cell-free to a whole-cell assay).
When moving from cell-free to whole cell assays, decreased potency is the more common outcome. As you mention, a big reason is just that you get a lot more things the drug can bind to, so you tend to have less drug effectively engaging the target. However, there are other factors at play as well. Ultimately, the main reason that one sees decreased potency in the whole-cell format is that one is often asking for something fundamentally different than what was asked for in the cell-free assay.
For example, let's say you do some CRISPR of RNAi screen in cancer cells, and you find that if you shut down gene X, the cells die. You look it up, and gene X codes for an enzyme.
So you think, "Great! I'll design an inhibitor for the enzyme, and then kill the cancer cells with that!"
In order to screen for inhibitors and test their effect on this enzyme, you set up an in vitro with purified enzyme, substrate, and some way of measuring the generation of product. You screen, find compounds, and optimize your lead. You eventually end up with a compound that has an IC50 of 1 nM in this assay. Then you decide it is time to move into cells. You treat the cancer cells with the compound, and you notice that your IC50 for cell death is 100 nM - 100x what it was in the original assay. The reason is that you are asking much more of your compound.
In the first assay, the compound had to bind to the enzyme, which may have been the only protein in solution, and inhibit it. You were directly measuring enzyme activity. In the second assay, the compound had to diffuse through the cell membrane or find some other means of getting into the cytosol, where the target protein resides. Then, despite the complex milieu in the cell, it had to bind that specific protein and inhibit it. But inhibiting the protein isn't enough. It has to inhibit the protein enough to kill the cell. Maybe the cells die when the enzyme activity is 50%, in which case you would expect your IC50 values to be fairly close between assays. But maybe it doesn't. You may need 70 or 80% inhibition before the cells start to actually die. Plus, the cells may not go down without a fight. Maybe they up-regulate expression of drug-efflux pumps, which are ATP-powered proteins that literally pump small molecules out of the cell. This reduces the effective concentration of drug you are giving the cells. There are also many other adaptations that the cells may undergo to improve survival, which are beyond the scope of this answer. So what you are demanding from your compound in the whole-cell assay may actually be much more than in the cell-free system.
The other effect - seeing increased potency in whole-cell vs cell-free conditions is much less common. Without seeing the exact study you are referring to, it is difficult to comment. However, one thing to remember about BACE1 is that it is part of a huge complex that is embedded in the cell membrane. Reconstituting this complex in vitro is non-trivial, and it may be that the complex is not functioning the same as in the cell depending on how the reconstitution was performed. For example, maybe BACE1 binds better to the drug only when a binding partner is present that stabilizes a conformation suitable for inhibitor binding. It could also be that the inhibitor is actually targeting another protein in the complex that is not present in the cell-free system.
Anyway, I apologize if this was a bit long, but I hope this answered your question and shed a bit of light on the complexities of assay development and drug development.
