Post Hoc Analysis

A post hoc analysis is any analysis conducted after data collection that was not specified in the original study protocol or statistical analysis plan. These analyses are considered exploratory rather than confirmatory because they are generated by patterns observed in the data rather than by prior hypotheses. While post hoc analyses can yield valuable insights, they carry an elevated risk of false positive findings.

The distinction between prespecified and post hoc analyses is central to research integrity and evidence interpretation. Prespecified analyses can control the overall error rate through predetermined hypotheses and planned statistical corrections. Post hoc analyses, by contrast, are vulnerable to data dredging, where researchers test multiple comparisons until they find something statistically significant. Journal editors, peer reviewers, regulatory agencies, and systematic reviewers all scrutinize post hoc findings more carefully than prespecified results, and many journals require explicit labeling of post hoc analyses. Transparent reporting of which analyses were planned and which emerged from the data is a cornerstone of credible research.

Post hoc analyses typically arise in several scenarios. A researcher may notice an unexpected pattern during data analysis and decide to investigate it further. Peer reviewers may request additional subgroup comparisons or sensitivity checks that were not in the original plan. Alternatively, after unblinding in a clinical trial, investigators may observe apparent differences in treatment response across demographic or clinical subgroups and pursue those leads statistically.

The core statistical concern is multiplicity. Each additional test performed on the same dataset increases the probability of finding at least one statistically significant result by chance alone. If a researcher conducts 20 independent tests at the 0.05 significance level and every null hypothesis is true, the probability of obtaining at least one false positive is approximately 64% (1 − 0.95^20). Post hoc analyses conducted without adjustments for multiple comparisons therefore inflate the overall Type I error rate well beyond the nominal alpha level.

Several strategies can mitigate this problem. Researchers should clearly label all post hoc results as exploratory and avoid drawing definitive conclusions from them. When possible, applying multiple comparison corrections such as the Bonferroni method or the Benjamini Hochberg procedure for controlling the false discovery rate can reduce the risk of spurious findings. However, the strongest approach is to treat post hoc findings as hypothesis generating and seek confirmation in independent datasets or prospectively designed follow up studies. Reporting guidelines do not forbid post hoc analyses but ask that they be distinguished: CONSORT 2025 (item 21d) asks trial reports to distinguish prespecified from post hoc additional analyses.

It is also important to recognize that not all post hoc analyses are inappropriate. A reviewer's request to adjust for an important confounder, or a sensitivity analysis testing whether results hold under different assumptions, can strengthen a paper even when not originally planned. The key issue is transparency: readers must be able to distinguish confirmatory results from exploratory ones. A post hoc analysis is also distinct from a post hoc test, such as Tukey's test after a significant ANOVA: those pairwise comparisons follow the overall test, but when they are named in the analysis plan in advance they are part of the prespecified analysis, not a post hoc analysis in this sense.

Post hoc analyses are appropriate when unexpected patterns in the data suggest important hypotheses worth investigating, as long as the analyses are labeled as exploratory. They are also appropriate when responding to reviewer requests for additional analyses. They should not be used to replace prespecified primary analyses, and they should never be presented as if they were planned from the outset. When a post hoc finding appears clinically important, researchers should design a new study with that specific hypothesis as the primary aim.

The most damaging mistake is presenting a post hoc finding as if it were prespecified, a practice sometimes called HARKing (hypothesizing after the results are known). This misleads readers into treating exploratory results as confirmatory evidence. A second frequent error is failing to adjust for multiple comparisons when reporting numerous post hoc tests, producing a long list of nominally significant p values that are largely attributable to chance. Researchers also sometimes perform post hoc analyses on underpowered subgroups, producing unreliable estimates with wide confidence intervals that are misinterpreted as evidence of differential treatment effects.

A randomized trial of a new antidepressant in 600 patients shows no significant difference in the primary outcome of Hamilton Depression Rating Scale scores at 12 weeks (mean difference 1.2 points, p = 0.14). During analysis, the investigators notice that patients with comorbid anxiety appear to benefit more. A post hoc subgroup analysis of 180 patients with comorbid anxiety yields a mean difference of 3.8 points (p = 0.009). The investigators report this as an exploratory finding and propose a new trial specifically enrolling patients with comorbid anxiety to test this hypothesis.