Predicting the optimal sample size for a nested experiment

Question

I need to test the effects of different treatments on plants and I would like to know the optimal sample size (number of plants that I need to sample). The variable I need to measure is the average number of insects present in a single plant. In the experiment, I have 3 main treatments and, in each treatment, I have 2 sub-treatments. The information I can provide are the following:

• Average number of larvae per plant: ranging between 0.03 and 0.1
• Standard deviation: ranging between 0.09 and 0.28
• The desired $Z$-score is 1.96
• I have no idea about the minimum detectable effect size but I can guess it could be 10%

Looking online, I have found different formulas that return different results. The first formula was:

$$n_1=\frac{(\sigma_1^2+\sigma_2^2/K)(z_{1-\alpha/2}+z_{1-\beta})^2}{\Delta^2},$$ where:

• $\Delta =|\mu_2-\mu_1|$ is the absolute difference between two means,
• $\sigma_1$, $\sigma_2$ are variances of mean #1 and #2,
• $n_1$, $n_2$ are sample sizes for group #1 and #2,
• $\alpha$ is the probability of type I error (usually 0.05),
• $\beta$ is the probability of type II error (usually 0.2),
• $z$ is the critical $Z$ value for a given $\alpha$ or $\beta$, and
• $k$ is the ratio of sample size for group #2 to group #1.

With this formula, comparing these two means (0.03 and 0.09) and using a standard deviation of 0.09, I am getting $n = 35$ for each treatment.

A different formula I found is:

$$n=\frac{2\sigma^2 Z^2}{d^2},$$ where

• $n$ is the required sample size per sub-treatment group,
• $\sigma^2$ is the estimated variance,
• $Z$ is the $Z$ score, and
• $d$ is the minimum detectable effect size.

With this formula, using a mean of 0.03 and a standard deviation of 0.09, I am getting $n > 6000$ for each treatment.

Could you suggest me a simple and efficient formula for calculating the sample size?

Answer 1

Power analyses depend on 4 numbers: alpha (false-positive threshold for significance testing), power (probability to reject the null hypothesis given all the other parameters), effect size (how big is the difference between groups, relative to the variability), and sample size.

Given 3 of these numbers, you can calculate the 4th.

If you want to calculate the sample size, you need to know what effect size you are designing your experiment to detect. This isn't optional or esoteric, it's absolutely critical to the power analysis. The effect size most often used for "t-tests" and similar is the ratio of the difference in means to the standard deviation. There are alternative forms if you expect the standard deviation to be different between groups.

You also need to choose your alpha and desired power. You haven't mentioned either of these in your post. There are some typical conventions, like alpha = 0.05, and power is usually at a minimum 0.80, though many would argue it should be higher, at least 0.90.

In your first calculation, you've set the difference in means to 0.06 and the standard deviation to be 0.09, so your effect size is 0.06/0.09 = about 0.67. This is often considered a moderate to large effect size.

In your second calculation, I don't know what exactly you've done, because you haven't written out any of your numbers, but I'm guessing this is where you've used your "10%" number. 10% of what, though? An effect size is not a percentage, it's a ratio. It can't be "10%". But if you've set it to 0.1, or 0.006, or something like that, this is a much smaller effect, and your power analysis should tell you that you need a much larger sample to detect this effect. Seems like that's what you got.

I'd recommend using a tool like G*Power to calculate sample size, but right now you are very lost and not understanding what you are undertaking, so I would really recommend to step back and sit down to read extensively about what power analysis is and means. This is not just a formula to plug in, this is critical to designing your experiment. Open up a statistics text book and don't move forward until you understand.

My preferred way to do power analysis is with simulation, because simulation lets you see what a power analysis is really doing and lets you understand more about your data. To simulate a power analysis, you generate synthetic data under the assumptions you have, like means and standard deviation for your different groups, then you actually do a statistical test on these data. Use your desired alpha threshold, run many many independent simulations (1000 to 100000 times), and note how often your p-value is smaller than this threshold. That's your power. You can change the sample size and see how the resulting power changes.

While running simulations, check whether the assumptions you've made about your data make sense! For example, if your mean is 0.03 and your standard deviation is 0.09, how often do you see negative numbers in your simulation? How often do you see non-integers in your simulation? Do negative numbers or non-integers make sense for a count of insects? No? Aha! Then you are violating some assumptions of the tests you're doing, and you probably shouldn't be doing t-tests with means and standard deviations, but rather using a statistical model that deals with count data.

If this all seems too complicated, it's time to consult with a statistician.