Fisher Exact Test

The Fisher exact test is a statistical test used to determine whether there is a significant association between two categorical variables in a 2 x 2 contingency table. Unlike the chi square test, which uses an approximation, the Fisher exact test calculates the exact probability of obtaining the observed distribution (or one more extreme) under the null hypothesis of no association, given the observed row and column totals. It is particularly appropriate when sample sizes are small.

Many research questions involve testing associations between two categorical variables when sample sizes are limited. The chi square test, the most commonly used alternative, relies on a large sample approximation that becomes inaccurate when expected cell frequencies are small (conventionally below 5). In these situations, the Fisher exact test provides a reliable p value by computing exact probabilities rather than relying on approximations. It is widely used in medical research, genetics, epidemiology, and any field where researchers encounter small sample sizes or rare events. Clinical trials often use it for safety analyses, where adverse events may be rare and standard asymptotic tests would be unreliable.

The Fisher exact test is based on the hypergeometric distribution. It considers all possible 2 x 2 tables that have the same row and column totals (marginals) as the observed table and calculates the probability of each under the null hypothesis of independence. The p value is the sum of the probabilities of all tables as extreme as or more extreme than the observed table. For a one sided test, "more extreme" means further in the specified direction. For a two sided test, the most common definition, used by R and SciPy, sums every table whose probability is no greater than that of the observed table; another convention doubles the one sided p value, so software can report different two sided p values for the same table.

For a 2 x 2 table with cells a, b, c, and d, and marginal totals R1 = a + b, R2 = c + d, C1 = a + c, and C2 = b + d, the probability of the observed table is given by the hypergeometric formula: (R1! × R2! × C1! × C2!) divided by (N! × a! × b! × c! × d!), where N is the total sample size. The test can be one sided (testing for association in a specified direction) or two sided (testing for any association).

For tables larger than 2 x 2, extensions of the Fisher exact test exist (such as the Freeman Halton test), though computational demands increase substantially with table size. Modern statistical software handles these computations efficiently, making the Fisher exact test practical even for moderately large tables.

The test is conservative, particularly with small samples. Because it conditions on both sets of margins, only a limited number of tables are possible, so the p value can take only a few distinct values and the actual Type I error rate is often well below the nominal alpha; it also tends to reject less often than the uncorrected chi square test. Some statisticians recommend the mid p value, which counts only half the probability of the observed table, to reduce this conservatism, at the cost of no longer guaranteeing that the Type I error rate stays at or below alpha.

The Fisher exact test should be used when analyzing 2 x 2 contingency tables with small sample sizes (by a common rule of thumb that goes back to Cochran, when any expected cell frequency is below 5), when analyzing rare events (such as adverse reactions in clinical trials), and when exact p values are needed rather than approximations. It is also appropriate as a default test for 2 x 2 tables in any sample size, since it remains valid at any sample size (its Type I error rate does not exceed the chosen alpha) and modern computers can handle the computation easily, although it is conservative. The chi square test remains more practical for larger tables (beyond 2 x 2) with adequate cell sizes.

A common error is using the chi square test when expected cell frequencies are very small, producing inaccurate p values, when the Fisher exact test would be appropriate. Conversely, some researchers believe the Fisher exact test can only be used with small samples, when in fact it is valid for any sample size. Another mistake is applying only a one sided test without justification; unless there is a strong a priori reason to expect the association in a particular direction, a two sided test should be used. Researchers sometimes report Fisher exact test results without also reporting the observed proportions or odds ratio, making the clinical or practical significance of the association impossible to evaluate. Finally, applying the Fisher exact test to data with more complex structures (such as matched pairs or repeated measures) is incorrect; these designs require other methods such as McNemar's test.

A clinical trial examines whether a new drug causes liver toxicity. Among 45 patients receiving the drug, 4 develop elevated liver enzymes (8.9%). Among 42 patients receiving placebo, 0 develop elevated liver enzymes (0%). The 2 x 2 table has very small expected cell frequencies (expected count for the drug plus liver toxicity cell is approximately 2.1), making the chi square test inappropriate. The Fisher exact test yields p = 0.12 (two sided). With only 4 patients having the event, no possible split of those events between the two arms could give a two sided p value below 0.05, which shows how conservative the test is when events are very few. Despite the numerical difference in proportions, the result is not statistically significant at the 0.05 level, although the investigators flag liver toxicity for monitoring in subsequent larger trials.