Methodological Quality

Methodological quality refers to the degree to which a study's design, conduct, and analysis minimize the risk of bias and support valid, reliable conclusions. It is assessed using standardized critical appraisal tools that evaluate specific aspects of the study's methodology, such as randomization, blinding, attrition, and selective reporting.

The conclusions drawn from any body of evidence are only as strong as the methods used to generate it. Studies with poor methodological quality may produce biased or unreliable results that, when incorporated into systematic reviews or clinical guidelines, lead to incorrect recommendations. Conversely, studies with high methodological quality provide a more trustworthy basis for decision making. Systematic reviewers, guideline developers, and HTA bodies routinely assess methodological quality to determine how much weight to give each study, whether to include or exclude studies from quantitative synthesis, and how to rate the overall certainty of evidence using frameworks such as GRADE.

Methodological quality is assessed using standardized tools tailored to specific study designs. For randomized controlled trials, the Cochrane Risk of Bias tool (RoB 2) evaluates five domains: the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of reported results. Each domain is rated as low risk, some concerns, or high risk of bias.

For observational studies, common tools include the Newcastle Ottawa Scale (NOS), which assesses selection, comparability, and outcome (in cohort studies) or exposure (in case control studies), and ROBINS-I, which in its 2016 version evaluates seven domains of bias in non randomized studies of interventions. For diagnostic accuracy studies, QUADAS-2 has been the standard tool; its successor, QUADAS-3, was published in February 2026 and is now the version its developers recommend. For qualitative studies, tools such as the Critical Appraisal Skills Programme (CASP) checklist address trustworthiness, credibility, and relevance.

The assessment process typically involves two independent reviewers who apply the chosen tool to each included study, with disagreements resolved through discussion or by a third reviewer. Results are reported in risk of bias tables or summary figures, and they inform decisions about subgroup or sensitivity analyses (for example, restricting the meta analysis to studies at low risk of bias).

It is important to distinguish methodological quality from reporting quality. A study may be well designed and conducted but poorly reported, making it impossible for reviewers to assess its true quality. Reporting guidelines such as CONSORT, STROBE, and PRISMA aim to improve reporting quality, while risk of bias tools assess the underlying methodology.

Assess methodological quality for every study included in a systematic review, evidence synthesis, or clinical guideline. The choice of assessment tool should match the study design. Quality assessments should be conducted independently by at least two reviewers, documented transparently, and used to inform the interpretation and synthesis of evidence. In meta analyses, sensitivity analyses excluding studies at high risk of bias test the influence of study quality on the pooled estimate. In GRADE assessments, risk of bias is one of five domains that can lower the certainty of evidence, alongside inconsistency, indirectness, imprecision, and publication bias.

A common error is using a quality assessment tool designed for one study type to evaluate a different type (for example, applying the NOS to a randomized trial). Another mistake is using quality scores as numerical summaries (for example, "studies scoring above 7 on a 10 point scale were included") rather than evaluating specific domains that are most relevant to the research question. Reducing quality assessment to a single number obscures which aspects of the study are strong and which are weak. Failing to conduct quality assessment independently (having only one reviewer) introduces assessor bias. Finally, excluding studies solely on the basis of quality without conducting and reporting sensitivity analyses limits the transparency and reproducibility of the review.

In a systematic review of pharmacological interventions for migraine prevention, two reviewers independently assess 30 included RCTs using the Cochrane RoB 2 tool. Of the 30 studies, 12 are rated as low risk of bias across all domains, 14 have some concerns (primarily related to deviations from intended interventions due to open label designs), and 4 are rated as high risk of bias (due to substantial missing outcome data and concerns about selective reporting). The primary meta analysis includes all 30 studies, while a sensitivity analysis restricted to the 12 low risk studies produces a slightly larger pooled effect, suggesting that the 18 trials with some concerns or at high risk of bias may have pulled the pooled estimate toward a smaller benefit.