Sensitivity Analysis

A sensitivity analysis is an analytical procedure that tests how the results of a study, model, or systematic review change when key assumptions, decisions, or parameters are varied. It assesses the stability of the findings and identifies which factors have the greatest influence on the conclusions, providing a measure of how much confidence can be placed in the primary results.

Every research study involves assumptions and methodological choices that could plausibly have been made differently. Sensitivity analysis examines whether the conclusions would change under alternative, reasonable assumptions. If the results remain consistent across different analytical choices, the findings are considered more trustworthy. If the results change substantially, the original conclusions should be interpreted cautiously, and the influential factors warrant further investigation. Sensitivity analysis is recommended by the Cochrane Handbook for meta analyses and expected in health economic evaluations submitted to health technology assessment bodies such as NICE, and reporting guidelines such as PRISMA 2020 and STROBE ask authors to describe any sensitivity analyses they conducted. Skipping sensitivity analysis does not itself bias a result, but it leaves readers unable to tell whether the conclusions depend on choices that could reasonably have been made differently.

Sensitivity analyses take different forms depending on the research context. In systematic reviews and meta analyses, common sensitivity analyses include excluding studies at high risk of bias, changing the statistical model (fixed effect versus random effects), using different effect measures (risk ratio versus odds ratio), excluding outlier studies, and varying the inclusion criteria (such as including or excluding studies published only as abstracts).

In health economic evaluations, deterministic sensitivity analysis varies one parameter at a time (one way sensitivity analysis) or multiple parameters simultaneously (multiway sensitivity analysis) to assess the impact on cost effectiveness conclusions. Tornado diagrams display the relative influence of each parameter. Probabilistic sensitivity analysis varies all uncertain parameters simultaneously by drawing values from their probability distributions, typically over thousands of model runs.

In observational studies, sensitivity analyses may examine the potential impact of unmeasured confounding. Quantitative bias analysis methods estimate how strong an unmeasured confounder would need to be to explain away the observed association. The E value, for example, quantifies the minimum strength of association that an unmeasured confounder would need to have with both the treatment and the outcome to reduce the observed effect to null.

Scenario analyses are a related technique in which entirely different analytical approaches or data configurations are tested, such as using intention to treat versus per protocol populations or including versus excluding specific data sources.

Conduct sensitivity analyses in every systematic review, meta analysis, and health economic evaluation. In observational studies, assess the potential impact of unmeasured confounding. In clinical trials, test the sensitivity of results to different analytical populations (intention to treat versus per protocol) and methods for handling missing data. Present sensitivity analysis results transparently, including cases where the conclusions change, not just those that confirm the primary analysis.

A common error is conducting only sensitivity analyses that support the primary findings and omitting those that challenge them. This selective presentation undermines the purpose of sensitivity analysis. Another mistake is failing to prespecify the planned sensitivity analyses in the protocol, making it difficult for readers to distinguish planned from ad hoc analyses. Researchers sometimes conduct sensitivity analyses but bury the results in supplementary materials without discussing their implications for the study's conclusions. Finally, testing only trivial or implausible variations (such as changing the alpha level from 0.05 to 0.049) does not provide meaningful information about the robustness of the findings.

A meta analysis of 24 trials finds that a cognitive intervention improves working memory scores by a standardized mean difference of 0.42 (95% CI: 0.28 to 0.56). The team conducts five prespecified sensitivity analyses: excluding 6 studies with high risk of bias (SMD = 0.31, still significant), using a fixed effect model (SMD = 0.38), excluding 3 studies with imputed data (SMD = 0.44), restricting to studies with active controls (SMD = 0.28), and excluding 2 outlier studies (SMD = 0.39). The pooled estimate ranges from 0.28 to 0.44 across analyses, supporting the stability of the finding, though the active control restriction substantially reduces the effect.