Risk of Bias

Risk of bias refers to the likelihood that the design, conduct, or analysis of a study has introduced systematic errors that could lead to an inaccurate estimate of the true effect. It is assessed using structured tools that evaluate specific domains, such as randomization, blinding, attrition, and outcome measurement, rather than providing a single global quality score.

Studies with high risk of bias may overestimate or underestimate treatment effects, leading to misleading conclusions. In systematic reviews, assessing risk of bias is essential for interpreting the reliability of individual study results and the overall body of evidence. The GRADE framework uses risk of bias as one of five domains for rating the certainty of evidence, and a body of evidence predominantly composed of high risk studies may be downgraded. Understanding risk of bias helps clinicians, policymakers, and researchers distinguish trustworthy evidence from unreliable findings.

Risk of bias is assessed using validated tools designed for specific study types. The Cochrane RoB 2 tool for randomized trials evaluates five domains: the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result. Each domain is rated as low risk, some concerns, or high risk, based on signaling questions that guide the assessment. The domain judgments then lead to an overall judgment for the result: low risk only if every domain is at low risk, and high risk if any domain is at high risk (or if some concerns in several domains together substantially lower confidence in the result). RoB 2 assesses a specific result, so different outcomes from the same trial can receive different judgments. ROBINS-I evaluates nonrandomized studies of interventions; its original 2016 version has seven domains, and a version 2, still marked as a draft in its November 2025 revision, reorganizes the assessment into six. ROBINS-E (2024) is the corresponding tool for nonrandomized studies of exposures. QUADAS-2, and its successor QUADAS-3, published in February 2026, assess diagnostic accuracy studies. The Newcastle Ottawa Scale evaluates observational studies.

The assessment is conducted by at least two independent reviewers for each included study. Disagreements are resolved through discussion or adjudication by a third reviewer. Results are presented in summary tables (showing the risk of bias judgment for each domain of each study) and graphical displays (such as traffic light plots).

Risk of bias results inform the synthesis in several ways. Sensitivity analyses can examine whether pooled results change when high risk studies are excluded. Subgroup analyses can compare effect estimates between low risk and high risk studies. The overall risk of bias profile influences the GRADE certainty rating for the body of evidence.

Assess risk of bias in every systematic review. Select the assessment tool that matches the included study designs. Conduct the assessment once the included studies are known and before the synthesis, and present the results transparently. Incorporate risk of bias findings into the interpretation of the synthesis results and the GRADE assessment. When designing primary studies, use risk of bias frameworks as a checklist to minimize the introduction of bias at each stage.

A common mistake is using a single composite quality score instead of domain level assessment, which obscures the specific sources of bias. Another error is assessing risk of bias with a single reviewer, introducing subjective inconsistency. Researchers sometimes rate a domain as "unclear" (in the original Cochrane tool) or answer signaling questions with "no information" (in RoB 2) without attempting to obtain missing information from study authors. Finally, presenting risk of bias results in a table without using them to inform the interpretation of the review's findings reduces the assessment to a formality.

A systematic review of 20 RCTs evaluating a behavioral intervention for smoking cessation uses Cochrane RoB 2. Of the 20 trials, 7 are rated low risk across all domains, 8 have some concerns (mainly regarding blinding, which is inherently difficult for behavioral interventions), and 5 are rated high risk (due to high attrition above 30% and evidence of selective outcome reporting). A sensitivity analysis restricted to the 7 low risk trials produces a smaller but still significant pooled odds ratio (1.8 versus 2.3 in the full analysis), suggesting that bias in the 13 trials with some concerns or at high risk may have inflated the apparent effect.