Dual Reviewer Screening

Dual reviewer screening is a methodological safeguard in systematic reviews in which two independent reviewers assess each study for inclusion at both the title and abstract screening stage and the full text screening stage. Disagreements between the two reviewers are resolved through discussion, consensus, or adjudication by a third reviewer. This process reduces the risk of erroneously excluding relevant studies and increases the reliability and transparency of the study selection process.

Study selection is one of the most consequential steps in a systematic review. Including or excluding a single study can change the pooled effect estimate in a meta analysis, alter the conclusions of a qualitative synthesis, or shift the certainty of evidence rating. Single reviewer screening has been shown to miss relevant studies: in one evaluation, single reviewers missed on average 8% of eligible reports (range 0 to 24%), whereas pairs of reviewers missed almost none, and in a randomized trial run on the Cochrane Crowd platform, single reviewer abstract screening missed 13% of relevant studies, against 3% for dual screening. These missed studies can introduce selection bias into the review. Cochrane's conduct standards make it mandatory for at least two people, working independently, to decide whether each study meets the eligibility criteria, and recommend, without requiring, that titles and abstracts also be screened in duplicate; Joanna Briggs Institute methodology also calls for two independent reviewers. PRISMA 2020 is a reporting guideline rather than a conduct standard: it asks authors to report how many reviewers screened each record and whether they worked independently. Reviewers and editors assess whether dual screening was performed when evaluating the quality of a submitted systematic review.

The process begins with the development of eligibility criteria, typically based on the PICO (Population, Intervention, Comparator, Outcome) framework, and a screening form that operationalizes these criteria into specific, answerable questions. Before screening begins, the review team conducts a calibration exercise in which all screeners independently assess the same sample of records and then compare decisions. Discrepancies are discussed, the eligibility criteria are clarified, and the screening form is refined. This calibration step helps all reviewers interpret the criteria consistently.

During title and abstract screening, each reviewer independently classifies every record as "include," "exclude," or "uncertain." Records that either reviewer includes or marks as uncertain proceed to full text screening, following the principle of being inclusive at this stage to avoid missing relevant studies. Automated screening tools and machine learning classifiers can assist with prioritizing records (presenting the most likely relevant records first), but final inclusion decisions rest with human reviewers, and the Cochrane Handbook does not currently recommend letting prioritization software exclude unscreened records automatically, because safe stopping rules still need to be developed and validated.

During full text screening, the two reviewers independently apply the eligibility criteria to the complete text of each potentially relevant study. At this stage, reasons for exclusion are recorded for each excluded study, and these reasons are reported in the PRISMA flow diagram. When the two reviewers disagree, they discuss the specific points of disagreement, refer to the eligibility criteria, and attempt to reach consensus. If consensus cannot be achieved, a third reviewer or the review team leader makes the final decision.

Inter rater reliability, typically measured using Cohen's kappa or percentage agreement, is often reported for one or both screening stages. On the conventional Landis and Koch benchmarks, kappa values of 0.61 to 0.80 indicate substantial agreement and values above 0.80 almost perfect agreement; many review teams treat a kappa above about 0.60 as acceptable, though these labels are conventions rather than rigid standards.

Dual reviewer screening is the standard for systematic reviews, including systematic reviews of interventions, diagnostic test accuracy reviews, qualitative evidence syntheses, and scoping reviews following the PRISMA-ScR extension. Cochrane makes independent duplicate decisions mandatory when deciding whether each study meets the eligibility criteria, and recommends duplicate screening of titles and abstracts. In rapid reviews, where time constraints are severe, a modified approach may be acceptable but should be acknowledged as a limitation. For example, the Cochrane Rapid Reviews Methods Group suggests that two reviewers screen at least 20% of titles and abstracts in duplicate, one reviewer screens the rest, and a second reviewer screens all excluded abstracts; at full text, one reviewer screens all articles and a second screens all excluded ones. Outside clearly labeled rapid reviews, leaving the final inclusion decisions to a single reviewer falls short of Cochrane's conduct standards.

A common mistake is conducting calibration once at the beginning and never revisiting it, even when disagreements suggest ongoing differences in interpretation. Another error is treating dual screening as a formality rather than an independent process, with reviewers screening side by side and influencing each other's decisions. Using overly vague eligibility criteria that cannot be applied consistently produces high disagreement rates and wastes reviewer time. Failing to record and report reasons for exclusion at the full text stage, which PRISMA 2020 asks authors to report (item 16b), reduces the transparency of the review. Some review teams resolve all full text disagreements by defaulting to inclusion, which inflates the number of studies proceeding to data extraction without truly resolving the eligibility question; the inclusive rule belongs at the title and abstract stage, not at the final eligibility decision. Finally, relying on machine learning screening without human verification of excluded records risks missing relevant studies that no person has seen.

A systematic review of nonpharmacological interventions for insomnia in older adults retrieves 6,834 records from five databases after deduplication. Two reviewers independently screen all titles and abstracts using Covidence, with a calibration exercise on the first 100 records achieving a kappa of 0.82. Title and abstract screening sends 312 records to full text review, including the 137 records (2.0%) on which the two reviewers disagreed, which go forward under the inclusive rule. Full text screening by both reviewers identifies 38 studies meeting all eligibility criteria, with 8 disagreements resolved by a third reviewer. The most common reason for exclusion at full text is wrong population (participants under age 60), documented for each excluded study in the PRISMA flow diagram.