Statistical Significance

A result is statistically significant when its p value, the probability of obtaining data at least as extreme as those observed if the null hypothesis were true, falls below a threshold set in advance (the alpha level), typically 0.05. Statistical significance indicates that data like these would be unusual if there were no true effect, but it does not show that the effect is real, clinically important, practically meaningful, or large in magnitude.

Statistical significance has been the dominant framework for drawing conclusions from quantitative research for much of the past century. It provides a standardized criterion for deciding whether observed differences or associations are larger than chance variation alone would usually produce if there were no true effect. However, widespread misinterpretation of p values and overreliance on the binary significant/not significant distinction have led to calls for reform, including the American Statistical Association's 2016 statement on p values, a 2019 editorial by the statement's two authors and a colleague advising that the term "statistically significant" no longer be used, and increasing emphasis on effect sizes, confidence intervals, and clinical significance alongside statistical significance.

Statistical significance testing follows a standard procedure. The researcher formulates a null hypothesis (typically that there is no difference or no association) and an alternative hypothesis. A test statistic is calculated from the data (such as a t statistic, chi square statistic, or z statistic), and the p value is computed as the probability of observing a test statistic as extreme as or more extreme than the one obtained, assuming the null hypothesis is true.

If the p value falls below the predetermined alpha level (typically 0.05), the null hypothesis is rejected and the result is declared statistically significant. If the p value exceeds alpha, the null hypothesis is not rejected (note: this is not the same as accepting the null hypothesis or proving that no effect exists).

The p value is often misinterpreted. It does not represent the probability that the null hypothesis is true, the probability that the result occurred by chance, or the probability that a replication study would find the same result. It is simply the probability of the observed (or more extreme) data under the assumption that the null hypothesis is true.

Statistical significance should be distinguished from clinical or practical significance. A very large study can detect a trivially small effect as statistically significant (a blood pressure reduction of 0.5 mmHg, for example), while a small study may fail to detect a clinically important effect because it lacks power. Reporting effect sizes and confidence intervals alongside p values provides a more complete picture.

Use statistical significance testing as one component of data interpretation, alongside effect sizes, confidence intervals, and clinical significance. Apply it to prespecified hypotheses in confirmatory analyses. Report exact p values rather than simply stating "p < 0.05" or "p = NS." Consider the clinical context when interpreting results: a statistically significant but clinically negligible finding may not warrant changing practice, while a clinically meaningful effect that narrowly misses statistical significance should not be dismissed outright.

A common error is equating statistical significance with practical importance, treating any p < 0.05 as evidence of a meaningful finding regardless of effect size. Another major mistake is interpreting a nonsignificant result as proof that no effect exists, ignoring the possibility that the study was simply underpowered. Researchers sometimes p hack (trying multiple analyses until one produces a significant result) or engage in HARKing (hypothesizing after the results are known), both of which inflate false positive rates. Finally, reporting only whether results were or were not significant, without providing exact p values, effect sizes, and confidence intervals, provides insufficient information for interpretation.

A randomized trial of 36,000 adults (18,000 per group) tests whether taking a dietary supplement for one year reduces the incidence of upper respiratory infections. The supplement group has an infection rate of 18.2% versus 19.1% in the placebo group (risk difference = 0.9 percentage points, 95% CI: 0.1 to 1.7, p = 0.03). Although the result is statistically significant, the absolute risk reduction of less than 1 percentage point is clinically negligible. The number needed to treat is 1 ÷ 0.009 = 111.1, rounded up to 112, meaning about 112 people would need to take the supplement for one year to prevent one infection. The researchers conclude that despite statistical significance, the supplement does not produce a clinically meaningful benefit.