Post Hoc Test

A post hoc test is a statistical procedure applied after an analysis of variance (ANOVA), usually one that yields a significant result, in order to identify which specific group means differ from one another. Because ANOVA only determines that at least one group is different without specifying which, post hoc tests perform pairwise comparisons while controlling for the increased risk of Type I errors that arises from multiple testing.

In any experiment comparing three or more groups, a significant ANOVA F test is only the first step. Without post hoc tests or planned comparisons, researchers cannot draw specific conclusions about which treatments outperformed others, which conditions produced different outcomes, or which groups diverged. Post hoc tests are therefore essential for translating a global finding of "some difference exists" into actionable, interpretable results. The choice of post hoc method directly affects both statistical power and error control, making it a methodological decision with real consequences for study conclusions.

After the ANOVA, usually once the F statistic is statistically significant, the researcher applies a post hoc procedure, ideally chosen in the analysis plan, to all pairwise comparisons (or a defined subset of comparisons). Each method handles the multiple comparison problem differently, balancing the tradeoff between controlling Type I error and maintaining statistical power.

Tukey's Honestly Significant Difference (HSD) is a widely used post hoc test for pairwise comparisons. It compares all possible pairs of group means simultaneously and controls the familywise error rate, meaning the probability of making at least one false positive across all comparisons remains at or below the nominal alpha level. Tukey's HSD works best with equal sample sizes across groups and assumes homogeneity of variance.

The Bonferroni correction is a simpler but more conservative approach. It divides the desired alpha level by the number of comparisons being made, so if a researcher performs 10 comparisons at alpha = 0.05, each individual comparison must reach p < 0.005 to be declared significant. While easy to apply and valid with unequal sample sizes, Bonferroni is overly conservative when the number of comparisons is large, increasing the risk of Type II errors.

The Scheffé test is the most conservative of the common post hoc methods and is best suited for testing complex contrasts (comparisons involving combinations of groups) rather than simple pairwise comparisons. Dunnett's test is specialized for comparing multiple treatment groups against a single control group, offering more power than Tukey or Bonferroni in that specific scenario. Games Howell is appropriate when the assumption of equal variances is violated, and it also handles unequal group sizes.

Tukey's HSD, Games Howell, Dunnett's test and Bonferroni corrected tests control the familywise error rate on their own, so a significant F test is not strictly required before them; many researchers still require one, which makes the overall procedure more conservative. Fisher's least significant difference (LSD) test is the exception: it relies on a significant F test for its protection, and even then it fully controls the familywise error rate only with three groups.

The choice among these methods depends on the research question, sample size balance, variance equality, and whether the comparisons are pairwise or involve contrasts. Researchers should select the method before analyzing the data and report the rationale in their methods section.

Use post hoc tests whenever ANOVA yields a significant result and you need to determine which groups differ. If you are comparing all groups to each other, Tukey's HSD is the standard choice. If you are comparing several treatments to one control, use Dunnett's test. If you need complex contrasts, consider Scheffé's test. A nonsignificant ANOVA does not in itself rule out a test such as Tukey's HSD, which controls the familywise error rate on its own, but decide in the analysis plan whether pairwise tests will be run whatever the F test shows; running them only after a disappointing F test, to look for something significant, is analytical flexibility. If you have specific, planned comparisons (such as treatment A versus placebo), use planned contrasts or a priori tests instead, which do not require a significant ANOVA.

A common error is performing multiple separate t tests instead of a proper post hoc procedure, which inflates the Type I error rate because it does not account for the total number of comparisons. For example, with five groups, there are 10 possible pairwise comparisons, and if those 10 t tests at alpha = 0.05 were independent, the chance of at least one false positive would be about 40%; because the comparisons share groups, the true figure is lower, roughly 28% when all five means are equal, but still far above 5%. Another mistake is selecting the post hoc method after seeing the data, choosing whichever test yields significant results, a form of analytical flexibility that undermines the validity of the findings. Researchers also sometimes decide to run post hoc tests only after a nonsignificant ANOVA, then report a single significant pair as the main finding; whether pairwise tests follow the F test, whatever its result, should be set in the analysis plan.

An educational psychology study compares reading comprehension scores across four instructional methods: traditional lecture, flipped classroom, peer teaching, and multimedia modules. ANOVA produces F(3, 196) = 5.14, p = 0.002. Tukey's HSD reveals that peer teaching produces significantly higher comprehension than traditional lecture (mean difference = 8.3 points, p = 0.001) and multimedia modules (mean difference = 5.7 points, p = 0.044), but the flipped classroom does not differ significantly from any other group.