Power Analysis

Power analysis is a statistical calculation performed during the planning stages of a study to determine the minimum sample size needed to detect an effect of a given magnitude with a specified level of confidence and statistical power. It links effect size, sample size, significance level, and power into a single framework that guides study design decisions.

Without a power analysis, researchers risk designing studies that are either too small to detect meaningful effects (underpowered) or unnecessarily large, wasting time and resources. Ethics committees and institutional review boards commonly expect a sample size justification, usually a power analysis, in study protocols, to help ensure that participants are not exposed to research procedures in studies unlikely to produce interpretable results. Grant reviewers scrutinize sample size justifications as a marker of scientific rigor. A well conducted power analysis also forces researchers to think carefully about the effect size they consider clinically or scientifically meaningful, a judgment that strengthens the entire study design.

A power analysis requires four inputs, and solving for any one of them is possible when the other three are specified. The most common approach is to fix the significance level (typically alpha = 0.05), the desired power (typically 80% or 90%), and the expected effect size, then solve for the minimum sample size.

The expected effect size is usually the most difficult parameter to specify. Researchers derive it from pilot studies, published literature on similar interventions, or the smallest effect they consider clinically important (the minimum clinically important difference). Effect sizes from small pilot studies are imprecise and may overestimate the true effect, so they should not be used on their own to set the sample size; basing the calculation on the smallest clinically important effect avoids this, and pilot data are more useful for estimating quantities such as the outcome's standard deviation. Meta analyses of prior work provide more stable estimates.

The choice of statistical test also affects the required sample size. A paired design (within subjects comparison) generally requires fewer participants than an independent groups design because it reduces variability. Similarly, more complex designs such as factorial experiments or studies with repeated measures have different power formulas. Software tools such as G*Power, R packages (pwr, simr), Stata, and SAS can compute sample sizes for a wide range of designs and statistical tests.

Researchers should also perform sensitivity analyses around the assumed effect size. Because the true effect is unknown before the study, computing sample sizes at several plausible effect sizes helps the research team understand the tradeoffs. If the study can only recruit a limited number of participants, a sensitivity analysis reveals the smallest effect the study can detect with adequate power, providing context for interpreting the eventual results. Finally, the computed sample size should be inflated to account for expected attrition, protocol violations, and incomplete data. A common approach is to divide by the expected completion rate (for example, dividing by 0.85 if 15% attrition is expected).

Conduct a power analysis during the protocol development phase, before data collection begins. A sample size justification is commonly expected in grant applications, clinical trial protocols, and ethics applications, and for confirmatory studies it is usually a power analysis. Power analysis is most informative for confirmatory studies with a clearly defined primary outcome. It is less applicable to exploratory, qualitative, or descriptive studies, though sample size justification is expected even in those designs. If published effect size estimates are unavailable, researchers can base the calculation on the smallest effect they consider clinically important, run a small pilot study to estimate quantities such as the outcome's standard deviation (rather than the effect size), or, for animal studies, use the resource equation method.

A frequent error is cherry picking an inflated effect size from a single small study to justify a conveniently small sample size. This produces a study that is actually underpowered for the true effect, which is likely smaller. Another mistake is failing to account for the study design in the power calculation, for example, using a formula for independent groups when the study uses a cluster randomized design, where the effective sample size is reduced by the design effect. Researchers also sometimes neglect to inflate for dropout, resulting in an underpowered study after attrition. Finally, performing a power analysis after the study is complete (post hoc power) is widely recognized as uninformative and should be avoided.

A team of pediatric researchers plans a cluster randomized trial to test whether a school based intervention reduces bullying. They expect a small to moderate effect size (Cohen's d = 0.35), set alpha at 0.05 and power at 0.80, and start from the sample size a two sample t test would need if students were randomized individually. G*Power yields a required sample of 130 students per arm (260 in total). Because the design randomizes schools, with an intraclass correlation of 0.05 and 20 students per school, the design effect is 1 + (20 − 1) × 0.05 = 1.95, inflating the required sample to 254 per arm (130 × 1.95 = 253.5, rounded up). Dividing by 0.90 to allow for 10% attrition gives 283 per arm, which at 20 students per school means 15 schools (300 students) per condition. The primary analysis then accounts for the clustering, for example with a mixed effects model, rather than a plain two sample t test.