Random Effects Model

A random effects model is a statistical framework used in meta analysis that assumes the true treatment effect varies across studies, rather than being identical. It accounts for both within study sampling error and between study variability (heterogeneity), producing a pooled estimate that represents the mean of a distribution of true effects rather than a single fixed true effect.

The choice between a fixed effect and random effects model is one of the most consequential methodological decisions in meta analysis. When the included studies differ in populations, interventions, settings, or methods, assuming a single fixed true effect is unrealistic. The random effects model accommodates this diversity by modeling the between study variance (tau squared) as an additional source of uncertainty. This leads to wider confidence intervals that more accurately reflect the uncertainty in the pooled estimate. Most contemporary meta analyses use random effects models because clinical and methodological diversity among studies is the norm rather than the exception.

In a random effects meta analysis, each study's effect estimate is treated as a draw from a distribution of true effects, characterized by a mean (the pooled effect) and a variance (tau squared); its square root, tau, is the standard deviation of the true effects on the scale of the effect measure. The model assumes that the observed effect in each study equals its true effect plus sampling error, and that the true effects themselves are normally distributed around the overall mean.

The between study variance (tau squared) is estimated from the data using methods such as DerSimonian and Laird (the most common, though known to underestimate tau squared with few studies), restricted maximum likelihood (REML), or the Paule Mandel estimator; REML is now the default in Cochrane's RevMan software. Once tau squared is estimated, the weight assigned to each study is a function of both its within study variance and the between study variance. This means that smaller studies receive a larger share of the total weight in a random effects model than in a fixed effect model, because the between study variance component diminishes the relative advantage of larger studies.

The pooled effect estimate and its confidence interval reflect both sources of uncertainty. When tau squared is large, the confidence interval widens, indicating greater uncertainty about the true mean effect. The prediction interval, which is distinct from the confidence interval, estimates the range within which the true effect in a new study or setting would likely fall, and is wider than the confidence interval when heterogeneity exists.

The I squared statistic quantifies the proportion of total variability attributable to between study heterogeneity rather than sampling error. Higgins and colleagues (2003) tentatively labeled values of 25%, 50% and 75% as low, moderate and high; the Cochrane Handbook gives overlapping ranges instead (0% to 40% might not be important, 30% to 60% moderate, 50% to 90% substantial, 75% to 100% considerable), and both are rough guides. However, I squared is influenced by the precision of the included studies and should be interpreted alongside tau squared and the prediction interval.

Use the random effects model when the included studies are expected to vary in their true effects due to differences in populations, interventions, settings, or methods, which is the case in most systematic reviews. The fixed effect model is appropriate when the included studies are functionally identical and there is no clinical or methodological reason to expect different effects, or when the aim is only to estimate the average effect for the specific set of studies included. When heterogeneity is negligible, the random effects model produces results similar to the fixed effect model. When in doubt, the random effects model is often the safer choice because it accounts for potential heterogeneity even if it happens to be minimal, with two caveats: with very few studies tau squared is estimated imprecisely, and because smaller studies gain weight, the random effects result is pulled further toward them if small study effects are present.

A frequent error is selecting between fixed effect and random effects models based on a statistical test for heterogeneity (such as Cochran's Q test), switching to random effects only when the test is significant. This approach is flawed because the Q test has low power with few studies and does not determine which model is conceptually appropriate. Another mistake is interpreting the random effects pooled estimate as if it applies uniformly to all settings, ignoring the between study variability captured by the prediction interval. Researchers also sometimes fail to report tau squared alongside I squared, leaving readers without a measure of the absolute magnitude of between study variability.

A meta analysis of 14 randomized trials evaluates the effect of mindfulness based stress reduction on anxiety. The random effects model yields a pooled standardized mean difference of 0.58 (95% CI: 0.39 to 0.77), with I squared = 43% and tau squared = 0.057 (so tau, the standard deviation of the true effects, is about 0.24). The 95% prediction interval is 0.02 to 1.14, indicating that while the average effect is moderate, a future study could find a nearly null effect or a very large effect depending on the setting. A sensitivity analysis using REML instead of DerSimonian and Laird produces a slightly larger tau squared estimate (0.08) and a wider confidence interval (0.37 to 0.79).