AMSTAR (A MeaSurement Tool to Assess systematic Reviews)

AMSTAR is a validated instrument designed to appraise the methodological quality of systematic reviews. It evaluates key domains including the comprehensiveness of the literature search, the appropriateness of study selection and data extraction procedures, the quality of risk of bias assessment, and the suitability of meta analytic methods when used.

Not all systematic reviews are created equal, and the reliability of a review's conclusions depends heavily on the rigor of its methods. AMSTAR provides a standardized framework for evaluating this rigor, enabling guideline developers, health technology assessment agencies, and researchers to distinguish high quality reviews from those with serious methodological limitations. Guideline developers, health technology assessment agencies, and authors of overviews of reviews commonly use AMSTAR when deciding how much to rely on a review. Without tools like AMSTAR, decision makers risk basing clinical guidelines on reviews whose conclusions may be unreliable due to incomplete searching, biased study selection, or inappropriate statistical methods.

The original AMSTAR instrument, published in 2007, contained 11 items covering domains such as whether an a priori design was provided, whether the literature search was comprehensive, whether study selection and data extraction were performed in duplicate, whether the quality of included studies was assessed and documented, whether appropriate methods were used to combine findings, and whether publication bias was evaluated.

AMSTAR 2, published in 2017, is the updated and improved version. It contains 16 items and distinguishes between critical and non critical domains. The seven critical domains address areas where methodological weaknesses are most likely to affect the validity of the review's conclusions: protocol registration, adequacy of the literature search, justification for excluding studies, risk of bias assessment of included studies, appropriateness of meta analytic methods, consideration of risk of bias when interpreting results, and assessment of publication bias. The AMSTAR 2 developers present these seven as a suggested set, and appraisers may justify adding or substituting critical domains for their purpose. The remaining nine items cover non critical domains that reflect best practices but are less likely to invalidate the review's conclusions.

Each item is rated "yes" or "no," and some items also allow "partial yes" when a standard is only partly met. Rather than adding up item scores, the overall rating is based on the pattern of weaknesses across critical domains, the overall confidence in the review's results is classified as high (no or one non critical weakness), moderate (more than one non critical weakness), low (one critical flaw with or without non critical weaknesses), or critically low (more than one critical flaw, with or without non critical weaknesses). The assessment is typically performed by two independent reviewers, with disagreements resolved by discussion or a third reviewer.

In the developers' test on 20 reviews, completing AMSTAR 2 took 15 to 32 minutes per review, not counting the time to read it. It can be applied to systematic reviews of healthcare interventions that include randomized or nonrandomized studies, or both, with or without meta analysis. Its developers did not intend it for diagnostic test accuracy reviews, individual participant data or network meta analyses, scoping reviews, or realist reviews, and it does not fit narrative reviews, which are not systematic.

Use AMSTAR 2 when evaluating the methodological quality of systematic reviews for inclusion in clinical guidelines, health technology assessments, or umbrella reviews (reviews of reviews). It is also useful for researchers who want to assess the quality of existing reviews in their field before designing a new review, helping to identify gaps in the evidence base. AMSTAR is not appropriate for appraising primary studies (use tools such as the Cochrane Risk of Bias tool, the Newcastle Ottawa Scale, or CASP checklists for those purposes).

A frequent error is applying AMSTAR to study types it was not designed for, such as narrative reviews, scoping reviews, or primary research studies. Each of these requires different quality appraisal instruments. Another mistake is treating the AMSTAR 2 overall rating as a simple sum of item scores; the tool uses a pattern based approach where a single critical flaw can reduce overall confidence to "low" regardless of performance on other items.

Reviewers sometimes rate items as "partial yes" when the information is simply not reported, conflating poor reporting with poor methodology. AMSTAR 2 guidance recommends rating items as "no" when the information needed to make a judgment is absent, because reviewers cannot assume that unreported steps were actually performed. Finally, failing to have two independent reviewers complete the AMSTAR assessment introduces the risk of subjective bias in quality ratings.

A guideline panel developing recommendations for the management of chronic low back pain identifies 15 systematic reviews published in the last five years. Two panel members independently apply AMSTAR 2 to each review. They find that four reviews receive a "high" confidence rating, six receive "moderate," three receive "low," and two receive "critically low." The panel excludes the two critically low reviews from the evidence base and notes the methodological limitations of the three low rated reviews when formulating their recommendations.