Is it necessary to conduct a power analysis before beginning an experiment?

Question

I know that power analysis is the statistically valid way to ensure you use the correct numer of samples or repeats in an experiment. But I have never seen any biologist actually conduct a power analysis. Mostly, researchers seems to use a rule of thumb (three technical, three biological replicates is a common one).

Should I be doing a power analysis each time I design an experiment, or can I just use one of the common biology rules of thumb? If not, what are the consequences for the validity of my results? And is there a situation where I will be required to use power analysis, which makes it advantageous to get used to doing it now?

Answer 1

You've already gotten a decent answer to this, but I'll provide my own thoughts on the subject.

Yes

It's necessary. It is absolutely something you should do before beginning an experiment, and preferably something you should do in collaboration with the person who is going to be helping you analyze your data. To address a couple points:

1. You'll see all kinds of researchers doing all manner of sloppy things when it comes to statistics and data analysis. Reading some journals makes me groan. You won't necessarily find it a problem for your field, but one should ask oneself if the goal is merely not to get called out by your peers, or to actually run a well designed experiment.
2. The consequences to the validity of your results lie in an increased risk of Type II error - the incorrect failure to reject the null hypothesis, or in slightly clearer English, finding no effect when an effect exists in reality. Which means, if you run an under powered study, you run the risk of doing the entire experiment, finding nothing, and being wrong. The consequences of that are myriad - first, it likely harms your chances to get published, as null results are often quite difficult to get into press. Second, if it does get into press, you've managed to get an incorrect finding into the literature, which will then be propagated in meta-analysis, reviews, and the minds of impressionable future readers. And then there's the chance you'll abandon a potentially productive line of inquiry because you couldn't be bothered to do power analysis.
3. One thing to also consider is that not having conducted power analysis somewhat limits your ability to chase after interesting sub-findings. If you've built your experiment on a shaky foundation, and then want to do a second analysis on a sub-set of your results, you almost certainly don't have the power for it.
4. There are times when you'll be required to do power analysis. If you do research that's clinically relevant, and you ever want it to appear in one of those journals, you may very well be required to show you had a properly powered experiment. And many grant applications require you do so. Even if they don't require it, some people will do a napkin math estimation of your power if they don't see it appearing in your application anywhere.

sjcockell is partially correct. To do power analysis, you at least need to have some notion of the effect measure you're likely to see. And these are indeed just estimations of what you'll see. But in nearly all circumstances, you'll likely have some ideas already. Are there similar experiments you can draw from? Your own pilot studies? A "feel" born of experience in your particular system?

It's also trivially easy to calculate power under a number of difference scenarios, to ensure your experiment is sufficiently powered if things go considerably worse than expected. For example, in a study I once did the power calculations for, we weren't sure what the ratio of exposed to unexposed subjects would be. So I ran it over a large range:

Which left me with the confidence that even if I was in my "worst case" scenario, I'd have reasonably good power at realistic effect sizes.

That's the true strength of power calculations. They'll tell you things about your study. What you need. What doesn't matter. Whether or not before you spend time and money pursuing an idea if you have a reasonable chance at success. Sit down with someone, take an hour or two (at most for a simple experiment) and do it right. Or ask CrossValidated for advice.

Answer 2

In order to calculate power, you need to know the variance of the data being collected. You can only estimate this prior to actually gathering the data itself, so any a priori power calculation is itself just an estimate.

This is why sometimes you will see small studies being conducted as pilot studies (for example, see (1)), which enable variance to be more accurately determined, and a true power calculation to be performed.

Knowing the power will enable you to state the probability that the effects you are observing are real (at a given sensitivity). The better idea you can get of this False Discovery Rate, the more solid the conclusions you can draw will be (so it is almost always a good idea to calculate it, but it is not always a feasible thing to do).

(1): Power and sample size estimation in microarray studies, Lin et al., BMC Bioinformatics 2010, 11:48.

Answer 3

Due to my own woeful ignorance on the subject, I have been reading up on statistical methods recently. From what (little) I understand, the real answer to this question is:

Yes, but only if you are doing Neyman-Pearson hypothesis testing

and

Absolutely not, if you are using Fisher p-values

That is, the question isn't formulated correctly, because power analysis is only valid under one statistical framework (Neyman-Pearson). And you are probably not using that framework.

In my experience, most experimental biologists use Fisher's p-value, which gives the probability of the data (or more extreme data) assuming that the null hypothesis is true. Under Fisher's framework, among other drawbacks, there is no quantitative measure of the test's power. However, it has the benefit that it allows scientists to do something close to what we would like to do--that is to draw conclusions from evidence obtained in individual experiments.

The Neyman-Pearson framework does included the idea of a test's power, because you must formulate an alternative hypothesis as well as desired alpha and beta error rates before starting your experiment. However, it mostly denies us the ability to make inferences from individual experiments, and for that reason appears less suited to experimental science. To quote from Goodman (see below), under Neyman-Pearson, "we must abandon our ability to measure evidence, or judge truth, in an individual experiment."

There is no clear right frequentist framework, although what is clear is that you cannot mix Fisher and Neyman-Pearson. Finally, although it doesn't really address your question directly, it seems wrong not to mention Bayesian methods as an alternative to these two frequentist frameworks, which comes with its own baggage.

Further reading from people that understand this much better than me:

Michael Lew's answer to "Setting the threshold p-value as part of hypothesis generation" at Cross Validated

Michael Lew's answer to "What are common statistical sins" at Cross Validated

Hubbard, Raymond, and M. J Bayarri. “Confusion Over Measures of Evidence ( p’S) Versus Errors (α’S) in Classical Statistical Testing.” The American Statistician 57, no. 3 (August 2003): 171–178. (Working Paper PDF)

Arguments for Bayesian statistics:

Goodman, Steven N. “Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy.” Annals of Internal Medicine 130, no. 12 (June 15, 1999): 995–1004.

Jaynes, E. T. Probability Theory: The Logic of Science (Online version of some parts)