Automated Screening

Automated screening refers to the use of machine learning, natural language processing, or other computational algorithms to assist in the screening of titles, abstracts, or full texts during systematic reviews. These tools prioritize or classify records based on their predicted relevance, reducing the manual workload required to identify eligible studies.

Systematic reviews often require screening thousands or even tens of thousands of records, a process that is labor intensive, time consuming, and costly. A typical systematic review team may spend hundreds of hours reading and classifying abstracts. In a 2015 systematic review of text mining approaches, most evaluations suggested that workload savings of between 30% and 70% might be possible, though sometimes at the cost of missing about 5% of relevant studies (a recall of about 95%); results vary by tool, topic and review. As the volume of published research keeps growing, with more than a million new records added to PubMed every year, manual screening becomes increasingly impractical, and automated tools are increasingly used to keep systematic reviews feasible within realistic timelines and budgets.

Automated screening systems typically operate through a process called active learning or supervised machine learning. The process begins with human reviewers screening a small initial batch of records manually and classifying them as relevant or irrelevant. These classifications serve as training data for a machine learning model, which learns to predict the relevance of unscreened records based on features extracted from their titles, abstracts, and metadata.

The model then ranks the remaining unscreened records by predicted relevance, presenting the most likely relevant records first for human review. As the reviewer screens additional records, the model is continuously updated with new training data, improving its predictions over time. This iterative process concentrates the reviewer's effort on the most promising records while deprioritizing those the model predicts are unlikely to be relevant.

Several approaches are used to determine when screening can stop. Conservative stopping rules aim to limit the share of relevant studies that remain undetected (a recall target of 95% or higher is commonly used). Some tools display a projected recall curve that estimates how many additional relevant studies are likely to be found by continuing to screen. The most cautious approach is to continue until the model predicts that the remaining unscreened records contain fewer than a specified number of relevant studies.

Tools with machine learning ranking or prioritization features include Abstrackr, Rayyan, ASReview, EPPI Reviewer, and Covidence, each with different algorithms and user interfaces. Some tools also incorporate text mining techniques to automatically extract data from included studies, further accelerating the review process.

Automated screening is appropriate for systematic reviews with large search results (often thousands of records) where manual screening of all records is feasible but time consuming. It is particularly valuable for rapid reviews and living systematic reviews that need to be updated regularly. It should not be used as a substitute for comprehensive searching, and human oversight of the screening process remains essential. Current guidance is that automated screening should assist, not replace, human judgment: people make the inclusion decisions, and any decision to stop screening before every record has been read needs a prespecified stopping rule that is reported with the review. The Cochrane Handbook does not currently recommend stopping screening early on the basis of software output, because safe stopping rules still need to be developed and validated.

A serious error is relying entirely on automated screening without human verification, which risks missing relevant studies that the algorithm fails to identify. Machine learning models are only as good as their training data, and a model trained on an unrepresentative initial sample may systematically miss certain types of relevant studies.

Another common mistake is failing to report the use of automated screening methods in the systematic review manuscript. Transparency about the tools used, the stopping criteria applied, and the estimated recall achieved is essential for readers to assess the completeness of the review. Researchers should also be cautious about using automated screening for reviews with very few relevant studies (low prevalence), where the model may have insufficient positive training examples to learn effectively. Finally, overconfidence in the model's predictions can lead reviewers to screen too few records manually, particularly in the later stages when the model predicts that most remaining records are irrelevant.

A systematic review team searching for studies on telehealth interventions for diabetes management retrieves 12,400 records from five databases. Using ASReview's active learning algorithm, two reviewers manually screen the first 500 records (identifying 48 as relevant) to train the model. The model then prioritizes the remaining 11,900 records. After screening 3,200 records total (26% of the initial set), the team has identified 312 relevant studies, and the model estimates 97% recall. The remaining 9,200 records are screened by a single reviewer to verify the model's predictions, confirming only 8 additional relevant studies. Because only the first 3,200 records were screened in duplicate, the team spends about 265 person hours in total, against an estimated 420 for screening every record in duplicate. Stopping at the model's 97% estimate without the single reviewer check would have taken about 110 person hours, but those 8 studies would have been missed.