Multiple Testing Correction

Multiple testing correction refers to a set of statistical adjustments applied when multiple hypothesis tests are performed simultaneously. These adjustments control either the probability of making one or more false positive errors (Type I errors) across the entire set of tests or the expected proportion of false positives among the significant results, because conducting many tests without correction inflates the overall chance of finding a "significant" result by chance alone.

Modern research frequently involves testing many hypotheses at once. Genome wide association studies test millions of genetic variants. Neuroimaging studies compare activity across thousands of brain regions. Clinical trials with multiple endpoints assess several outcomes simultaneously. Without correction, the probability of at least one false positive increases rapidly with the number of tests. If each test uses alpha = 0.05 and 20 independent tests are performed, the probability of at least one false positive is 1 minus (0.95)^20 = 0.64, or 64%. Multiple testing correction restores statistical rigor by adjusting the significance threshold or the reported p values to maintain the desired error rate.

Multiple testing corrections fall into two broad categories based on the error rate they control. Family wise error rate (FWER) methods control the probability of making one or more false positives across all tests. The Bonferroni correction is the simplest and among the most conservative, dividing the significance level (alpha) by the number of tests. For 20 tests at alpha = 0.05, the adjusted threshold is 0.05/20 = 0.0025. The Holm method is a sequential version that is less conservative while still controlling FWER: it orders p values from smallest to largest and compares each to a progressively less stringent threshold.

False discovery rate (FDR) methods control the expected proportion of false positives among all significant results, rather than the probability of any false positive. The Benjamini Hochberg procedure is the most common FDR method, and it is less conservative than FWER methods, making it more appropriate for exploratory analyses where some false positives are acceptable as long as the proportion is controlled. For 100 tests with FDR controlled at 0.05, the expected proportion of false positives among the significant results is held at or below 5%; in any one analysis the actual proportion can be higher or lower.

The choice between FWER and FDR depends on the research context. FWER control is preferred in confirmatory studies where any false positive is problematic (such as clinical trial primary endpoints). FDR control is preferred in discovery oriented studies where a moderate false positive rate is acceptable (such as genomic or proteomic screening).

Other approaches include permutation based methods, which estimate the null distribution of test statistics by randomly shuffling the data and comparing the observed statistics to this empirical distribution. These methods naturally account for the correlation structure among tests and can be more powerful than Bonferroni in settings where tests are not independent.

Apply multiple testing correction whenever you are conducting multiple hypothesis tests on the same dataset and the risk of false positives across the entire set is a concern. This includes subgroup analyses, multiple endpoint analyses, genome wide scans, and any analysis that involves more than a few predetermined comparisons. Do not apply multiple testing correction to a single prespecified primary analysis in a clinical trial, because a single test creates no multiplicity. When the analysis is exploratory, clearly label the results as hypothesis generating and report both adjusted and unadjusted p values.

A common error is conducting many comparisons without any correction and then presenting the "significant" results as confirmed findings, which becomes p hacking when the other comparisons go unreported. Another mistake is applying the Bonferroni correction in situations where it is overly conservative (such as when tests are highly correlated), reducing power to unacceptable levels. Some researchers apply corrections selectively, adjusting only when convenient or after seeing the results, which undermines the purpose of the correction. Failing to define the "family" of tests (which tests belong together for correction purposes) can lead to under or over correction. Finally, some researchers misinterpret FDR adjusted p values as FWER adjusted, leading to incorrect claims that the probability of even one false positive among the findings is controlled.

A clinical trial evaluates a new cancer immunotherapy on five prespecified endpoints, ranked in order of importance: overall survival, progression free survival, objective response rate, duration of response, and quality of life. The protocol specifies a hierarchical testing procedure in which overall survival is tested first at alpha = 0.05. If significant, progression free survival is tested at alpha = 0.05, and so on. If any test fails to reach significance, subsequent tests are not formally tested. This gatekeeping approach controls the family wise error rate at 0.05 across the five endpoints without dividing alpha five ways, as a Bonferroni correction would (0.05/5 = 0.01 per endpoint).