p Value

The p value is the probability of obtaining a result at least as extreme as the one observed in a study, assuming that the null hypothesis is true. It quantifies the compatibility of the observed data with a specified statistical model, most commonly the null hypothesis of no effect or no difference.

The p value is arguably the most widely used and most widely misunderstood quantity in all of research. It appears in virtually every discipline that uses quantitative methods, serving as the primary criterion for determining statistical significance and, by extension, influencing publication decisions, clinical guidelines, and policy. The conventional threshold of p < 0.05, proposed by Ronald Fisher in the early twentieth century and subsequently codified into standard practice, divides results into "significant" and "not significant," shaping careers, funding decisions, and the direction of entire fields. Yet the p value is also the source of profound methodological debate. Misinterpretation and misuse of p values have contributed to the replication crisis, to publication bias, and to the propagation of false positive findings. Understanding what the p value does and does not tell us is essential for every researcher.

The p value is calculated within the framework of null hypothesis significance testing (NHST). The process begins with stating a null hypothesis (H0), which typically posits no effect, no difference, or no association. A test statistic is computed from the observed data (for example, a t statistic, chi square statistic, or F statistic), and the p value is derived as the probability of obtaining a test statistic at least as extreme as the observed one under the assumption that H0 and the test's other assumptions are true.

If the p value is small (conventionally below 0.05), the data are judged incompatible enough with the null hypothesis for the null to be rejected in favor of an alternative hypothesis. If the p value is large, the data do not provide sufficient evidence to reject the null hypothesis, though this does not mean the null hypothesis is true. The p value is a continuous measure of how compatible the data are with the null hypothesis, and the binary threshold of 0.05 is a convention, not a natural law. A p value of 0.049 and a p value of 0.051 represent essentially the same degree of evidence, despite falling on opposite sides of the conventional cutoff.

Several critical clarifications about p values are necessary. The p value is not the probability that the null hypothesis is true. A p value of 0.03 does not mean there is a 3% chance that the result occurred by chance or a 97% chance that the effect is real. The p value does not measure the size or importance of an effect; a highly significant p value (p < 0.001) can accompany a trivially small effect in a large sample, while a nonsignificant p value can accompany a large and clinically important effect in a small sample. The p value is not a measure of replicability; a p value of 0.05 does not mean that the finding will replicate 95% of the time.

Because of these widespread misunderstandings, the American Statistical Association (ASA) issued an unprecedented formal statement in 2016 articulating six principles about p values, including that p values do not measure the probability that the studied hypothesis is true, that scientific conclusions should not be based solely on whether a p value crosses a specific threshold, and that proper inference requires full reporting and transparency. Many journals now require reporting of effect sizes and confidence intervals alongside p values, moving the field toward a more nuanced approach to statistical evidence.

Report p values as part of a comprehensive presentation of statistical results that includes effect sizes, confidence intervals, and sample sizes. Use them as one piece of evidence rather than the sole determinant of a finding's importance. Prespecify your significance threshold and adjust for multiple comparisons when testing multiple hypotheses simultaneously. Consider reporting exact p values (for example, p = 0.032) rather than inequalities (p < 0.05) to provide more information.

A common mistake is interpreting the p value as the probability that the null hypothesis is true or that the result occurred by chance. This reflects a fundamental misunderstanding of conditional probability. Another common error is equating statistical significance with practical or clinical significance; a statistically significant result may be too small to matter in practice, and a nonsignificant result may reflect insufficient statistical power rather than absence of an effect. p Hacking, the practice of selectively analyzing data or running multiple tests until a significant p value is obtained, inflates the false positive rate and is a major contributor to irreproducible findings. Finally, dichotomizing results as "significant" or "not significant" obscures the continuous nature of the evidence and leads to absurd distinctions between p = 0.049 and p = 0.051.

A nutritional epidemiology study examines whether daily consumption of green tea is associated with lower risk of cardiovascular disease in a cohort of 50,000 adults followed for 15 years. The hazard ratio for the association between daily green tea consumption and cardiovascular events is 0.88 (95% CI: 0.79 to 0.98, p = 0.02). The researchers report that the association is statistically significant but note that the effect size is modest (a 12% lower hazard of cardiovascular events), that residual confounding from lifestyle factors cannot be excluded, and that the finding should be interpreted alongside the full confidence interval rather than the p value alone.