Prediction Interval

In meta analysis, a prediction interval is a range that estimates where the true effect in a new study drawn from the same population of studies would be expected to fall. Unlike the confidence interval for the pooled estimate, which addresses uncertainty about the average effect, the prediction interval accounts for both within study sampling error and between study variability (heterogeneity), providing a more complete picture of the expected range of effects across different settings. Outside meta analysis, for example in regression, a prediction interval is a range for a single new observation, which is wider than the confidence interval for the mean because it also includes the scatter of individual values.

Clinicians, policymakers, and guideline developers need to know not just the average treatment effect but also how much the effect is likely to vary from one setting to another. A meta analysis might report a significant pooled effect with a narrow confidence interval, suggesting a reliable finding. However, if between study heterogeneity is substantial, the prediction interval may be wide and cross the null, indicating that the intervention may not be effective in every future setting. This distinction is critical for decisions about implementing interventions in specific populations or healthcare systems. Methodologists have recommended prediction intervals as an important supplement to confidence intervals in random effects meta analyses; IntHout and colleagues (2016), for example, argued that they should be presented routinely.

The prediction interval in a random effects meta analysis is calculated using the pooled effect estimate, the between study variance (tau squared), and the standard error of the pooled estimate. The formula resembles that of a confidence interval but adds the between study variance component. Specifically, in the method of Higgins, Thompson and Spiegelhalter (2009), the prediction interval is approximately equal to the pooled estimate plus or minus a t distribution critical value with k minus 2 degrees of freedom (k being the number of studies) multiplied by the square root of the sum of the squared standard error and tau squared (the Cochrane Handbook's simple version of the formula uses k minus 1 degrees of freedom, which gives a slightly narrower interval).

When heterogeneity is negligible (tau squared approaching zero), the prediction interval converges toward the confidence interval, because the between study variance contributes little additional uncertainty. As heterogeneity increases, the prediction interval widens, reflecting greater uncertainty about what any single future study would find. In extreme cases, the confidence interval may exclude the null (suggesting a statistically significant average effect) while the prediction interval includes the null (suggesting that some future studies may find no effect or even a harmful effect).

The number of studies in the meta analysis affects the precision of the prediction interval. With few studies, prediction intervals are estimated imprecisely: the tau squared estimate itself carries substantial uncertainty, and the interval rests on the assumption that the true effects are normally distributed, so it can look spuriously wide or spuriously narrow. The Cochrane Handbook encourages their use when the number of studies is reasonable (for example, five or more) and there is no clear funnel plot asymmetry; with fewer studies they should be interpreted with appropriate caution. Some methodological work has proposed modified prediction interval formulas for small numbers of studies.

Prediction intervals can be visualized on forest plots as a horizontal band extending beyond the confidence interval diamond, providing an intuitive display of the expected range of effects. Reporting both intervals gives readers a two layered understanding: the confidence interval for the average and the prediction interval for the individual study level.

Adding studies narrows the confidence interval, because the standard error of the pooled estimate shrinks, but the prediction interval cannot become narrower than the spread of the true effects themselves: with many studies, a 95% prediction interval approaches the pooled estimate plus or minus about 1.96 times tau. The same logic applies in regression, where the term is used more generally. A confidence interval for the mean outcome at given values of the predictors reflects only uncertainty in the fitted line, whereas a prediction interval for one new observation adds the residual variance: approximately the predicted value plus or minus a t critical value (with n minus 2 degrees of freedom in simple linear regression) multiplied by the square root of the residual variance plus the squared standard error of the fitted mean. A larger sample shrinks the confidence interval toward zero width, but the prediction interval stays at least as wide as the scatter of individual observations around the line.

Report prediction intervals in any random effects meta analysis where between study heterogeneity is present. They are particularly informative when decision makers need to know the range of plausible effects for implementation in a specific context. Prediction intervals are less useful in fixed effect meta analyses, where the assumption is that all studies share a common true effect. They are also less informative when heterogeneity is minimal, as they will closely approximate the confidence interval.

A common mistake is omitting prediction intervals entirely and relying solely on the confidence interval to characterize the results of a random effects meta analysis. This gives readers an incomplete picture, as the confidence interval describes only the uncertainty about the average effect, not how much the true effects vary. Another error is interpreting a prediction interval that crosses the null as evidence that the intervention does not work, when it actually means that the true effect may be absent, or even reversed, in some settings while beneficial in others. Researchers also sometimes confuse prediction intervals with confidence intervals, describing them interchangeably when they address categorically different questions.

A meta analysis of 18 randomized trials finds that a structured exercise program reduces HbA1c in patients with type 2 diabetes by an average of 0.45 percentage points (95% confidence interval: 0.30 to 0.60, p < 0.001). However, heterogeneity is substantial (I squared = 72%, tau squared = 0.09). The 95% prediction interval, computed with a t distribution on 16 degrees of freedom (18 trials minus 2), is negative 0.21 to 1.11, meaning that while the average effect is beneficial, the true effect in a new setting could range from a small rise in HbA1c of 0.21 percentage points to a reduction of 1.11 percentage points. The confidence interval excludes zero, but the prediction interval does not.