Funnel Plot

A funnel plot is a graphical tool used in meta analyses to visually assess whether publication bias or other small study effects may be present. It is a scatter plot in which each study's effect estimate is plotted on the horizontal axis against a measure of its precision (typically the standard error; the inverse of the standard error is also used) on the vertical axis. In the absence of bias, the points should form a symmetrical inverted funnel shape.

Publication bias is one of the most serious threats to the validity of meta analyses. Studies with statistically significant or positive results are more likely to be published, while studies with null or negative results may remain in file drawers. If a meta analysis includes only published studies, the pooled estimate may overestimate the true effect. Funnel plots provide a simple, visual method for detecting this problem. The Cochrane Handbook describes how to draw and interpret them, and PRISMA 2020 asks reviews to report how they assessed the risk of bias due to missing results, for which a funnel plot is one common method. While funnel plots cannot prove that publication bias exists (asymmetry can have other causes), they can raise a red flag that warrants further investigation.

In a standard funnel plot, the horizontal axis displays the effect measure (such as the odds ratio, risk ratio, or standardized mean difference) and the vertical axis displays a measure of study precision. The most common choice for the vertical axis is the standard error, plotted in reverse so that larger (more precise) studies appear at the top and smaller (less precise) studies appear at the bottom.

In the absence of publication bias, the plot should resemble an inverted funnel. Large studies cluster near the top with effect estimates close to the pooled estimate. Smaller studies, which have greater sampling variability, scatter more widely at the bottom but should be symmetrically distributed around the pooled estimate. The expected spread at each level of precision can be shown by plotting a triangular region bounded by the 95% confidence limits around the pooled estimate.

Asymmetry in the funnel plot suggests potential bias. If the plot shows a gap in the lower left (for a benefit measure) or lower right corner, it may indicate that small studies with null or unfavorable results are missing, consistent with publication bias. However, asymmetry can also result from genuine heterogeneity (small studies differ from large ones in population or intervention), poorer methodological quality in smaller studies, chance (especially with fewer than 10 studies), or choice of effect measure.

Statistical tests for funnel plot asymmetry (such as Egger's test, Begg's test, and the Peters test) complement the visual assessment by providing a formal test of whether asymmetry is statistically significant. Trim and fill analysis can estimate the number and location of missing studies and produce an adjusted pooled estimate.

Funnel plots should be constructed whenever a meta analysis includes 10 or more studies (with fewer studies, the visual assessment is unreliable and statistical tests have very low power). They can be used in many types of meta analysis, but in diagnostic test accuracy reviews standard funnel plots and asymmetry tests can be seriously misleading, so an adapted funnel plot based on effective sample size, with its own regression test (Deeks' test), is used instead. Funnel plots are most informative when interpreted alongside heterogeneity statistics, subgroup analyses, and formal statistical tests for asymmetry. When a meta analysis has 10 or more studies, they should be included as a standard figure in the publication.

A common error is interpreting funnel plot asymmetry as definitive proof of publication bias, when it is only suggestive and can have other causes. Another mistake is creating funnel plots with fewer than 10 studies, where visual assessment is unreliable and no pattern can be meaningfully interpreted. Using the wrong axis (for example, plotting sample size instead of standard error) can alter the expected shape and lead to incorrect interpretation. Some researchers create funnel plots but do not interpret or discuss them, treating them as a formality rather than a diagnostic tool. Finally, failing to consider whether the effect measure itself might explain the asymmetry (some measures, like odds ratios, have mathematical properties that can create apparent asymmetry) is a common oversight.

A meta analysis of 18 randomized trials examines the effect of a dietary supplement on cholesterol levels. The forest plot shows a pooled mean difference of minus 12.5 mg/dL (95% CI: minus 16.8 to minus 8.2). The funnel plot reveals a gap in the lower right quadrant: several small studies show large reductions in cholesterol, but no small studies show small or null effects. Egger's test confirms statistically significant asymmetry (p = 0.02). The trim and fill analysis estimates that 4 studies may be missing; the adjusted pooled estimate (minus 9.1 mg/dL) is smaller than the original, suggesting that publication bias may have inflated the effect.