Risk of Bias Assessment

Risk of bias assessment is the structured, systematic evaluation of the potential for systematic errors in a study's design, conduct, analysis, or reporting that could lead to over or underestimation of the true effect. It is a core component of systematic review methodology, applied to each included study using validated domain based tools to determine the trustworthiness of the study's findings.

Not all studies contribute equally reliable evidence to a systematic review. Risk of bias assessment provides the framework for distinguishing studies that are likely to produce valid results from those that may be compromised by methodological flaws. This distinction affects how evidence is synthesized, interpreted, and translated into recommendations. Without risk of bias assessment, a systematic review treats all studies as equally trustworthy, potentially giving undue weight to flawed evidence. Failing to take risk of bias into account can let systematic errors in the included studies distort the review's pooled estimates and overstate the certainty of its conclusions.

Risk of bias assessment follows a structured protocol. The reviewer selects an appropriate tool based on the study design of the included studies. For each study (or, with RoB 2, for each result being assessed), the reviewer answers a series of signaling questions within each domain, gathers information from the study report (and supplementary sources when available), and makes a judgment about the risk of bias for that domain.

The process requires at least two independent assessors to reduce subjective bias. Calibration exercises, in which assessors practice on a sample of studies and compare their ratings, improve consistency. A detailed description of the assessment process, including the tool used, the criteria for each judgment, and the procedure for resolving disagreements, should be included in the review protocol.

Risk of bias assessment results are typically presented in two formats. A summary table shows the risk of bias judgment for each domain of each study, allowing readers to identify patterns (such as a domain where most studies have high risk). A graphical display, such as a bar chart showing the proportion of studies at each risk level across domains, provides an overview of the evidence quality.

The results inform the synthesis through several mechanisms. At the study level, high risk of bias may lead reviewers to exclude certain studies in sensitivity analyses or to present results stratified by risk of bias. At the body of evidence level, the overall risk of bias profile contributes to the GRADE assessment of certainty.

Conduct risk of bias assessment in every systematic review, regardless of the topic or study design. The assessment should be completed after full text screening, once the included studies are known, and before the synthesis is conducted. Use the tool appropriate to the study design: RoB 2 for randomized trials, ROBINS-I for nonrandomized studies of interventions, QUADAS-2 or its successor QUADAS-3 (published in February 2026) for diagnostic accuracy studies, and validated tools for other designs. Document the assessment process in the protocol and present results transparently in the review.

A common error is applying a risk of bias tool without training, leading to inconsistent or inaccurate ratings. Another mistake is assessing risk of bias at the study level using a tool designed for outcome level assessment (RoB 2 is outcome specific, meaning different outcomes from the same study may have different risk of bias ratings). Researchers sometimes conflate risk of bias with methodological quality, using tools that combine bias assessment with reporting quality items. Finally, conducting risk of bias assessment but not incorporating the results into the synthesis or interpretation is a missed opportunity that reduces the value of the systematic review.

A Cochrane review of 32 trials of a pharmacological intervention for chronic pain conducts risk of bias assessment using RoB 2. The assessment reveals that while randomization and allocation concealment are adequate in most trials, 22 of 32 trials have high risk of bias in the "missing outcome data" domain because more than 20% of participants had no outcome data and dropout was more common among participants whose pain had not improved, so whether data were missing could depend on the outcome itself. The review team conducts a primary analysis including all trials and a sensitivity analysis restricted to the 10 trials with low risk across all domains. The sensitivity analysis produces a smaller effect estimate, and the authors note this difference when formulating their conclusions and GRADE assessment.