Quality Assessment

Quality assessment is the evaluation of individual studies included in a systematic review for their methodological rigor, examining potential sources of bias, confounding, and reporting deficiencies that could affect the validity and reliability of their findings. It is also referred to as critical appraisal or methodological quality assessment and is a standard step in the systematic review process, mandatory in Cochrane intervention reviews.

Not all studies contributing to a systematic review are created equal. Differences in study design, execution, and reporting can lead to varying degrees of bias, meaning that some studies provide more trustworthy estimates than others. Quality assessment allows reviewers to identify these differences, interpret results in context, and make informed decisions about how to weight or stratify evidence in the synthesis. Without quality assessment, a meta analysis weights each study by its precision alone, so a large but poorly conducted study can dominate the pooled estimate however biased it is, producing misleading results. Quality assessment results also feed into evidence grading frameworks such as GRADE, where risk of bias is a key domain for determining certainty of evidence.

Quality assessment is conducted using standardized tools appropriate to the study designs included in the review. For randomized controlled trials, the Cochrane Risk of Bias tool (RoB 2) evaluates domains including the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of reported results. For nonrandomized studies of interventions, ROBINS-I assesses confounding, participant selection, intervention classification, deviations, missing data, outcome measurement, and reporting (the domains of its 2016 version; a version 2, still marked as a draft in its November 2025 revision, drops the deviations domain and reorganizes the assessment into six domains). For diagnostic accuracy studies, QUADAS-2 has long been the standard tool; its successor, QUADAS-3, was published in February 2026. For observational studies, the Newcastle Ottawa Scale evaluates selection, comparability, and exposure or outcome assessment.

The assessment process typically involves at least two independent reviewers who evaluate each study against the tool's criteria and then compare their judgments. Disagreements are resolved through discussion or by consulting a third reviewer. Results are presented in summary tables or graphical displays (such as traffic light plots for the Cochrane RoB tool) that allow readers to see the quality profile of the included evidence at a glance.

Quality assessment results can be incorporated into the synthesis in several ways. Sensitivity analyses can compare results with and without high risk studies. Subgroup analyses can stratify by quality level. In meta analysis, quality scores can be used as weights, though this approach is controversial because composite scores may obscure the specific sources of bias. The GRADE framework uses risk of bias, judged from these assessments, as one of five domains for rating down the overall certainty of a body of evidence, downgrading when a substantial proportion of included studies are at high risk of bias.

Conduct quality assessment in every systematic review, regardless of the type of studies included. Select the assessment tool based on the study design: RoB 2 for randomized trials, ROBINS-I for nonrandomized intervention studies, QUADAS-2 or its successor QUADAS-3 for diagnostic accuracy studies, and other validated tools for observational, qualitative, or economic studies. Quality assessment should be done after full text screening, once the included studies are known, and the results should inform both the narrative synthesis and any quantitative analyses.

A common error is using a quality assessment tool designed for one study type on a different study type (such as applying the Newcastle Ottawa Scale to randomized trials). Another mistake is conducting quality assessment with a single reviewer, which introduces the risk of subjective bias in the ratings. Researchers sometimes report quality assessment results in a table but fail to incorporate them into the interpretation of the review's findings, treating the assessment as a box checking exercise. Finally, using composite quality scores (which sum individual items into a single number) can mask important domain specific weaknesses; domain level assessment is generally preferred.

A systematic review of physical therapy for low back pain includes 34 randomized trials and 12 observational cohort studies. Two reviewers independently assess the trials using the Cochrane RoB 2 tool and the cohort studies using the Newcastle Ottawa Scale. Among the randomized trials, 18 are rated low risk of bias across all domains, 10 have some concerns (primarily in the measurement of the outcome domain, because outcome assessors were not blinded), and 6 are rated high risk (due to high attrition and incomplete outcome data). A sensitivity analysis restricted to the 18 low risk trials finds a smaller but still significant effect compared to the overall pooled estimate, suggesting that methodological weaknesses in some trials may have inflated the apparent benefit.