Automation Bias

Automation bias is the tendency of human decision makers to over rely on the output of automated systems, including artificial intelligence tools, even when contradictory information is available from other sources or from the decision maker's own judgment. In research contexts, automation bias may affect processes such as systematic review screening, data analysis, and clinical decision support.

As automated tools become more prevalent in research workflows, from machine learning assisted screening in systematic reviews to AI driven data analysis and writing assistance, the risk of automation bias grows. Researchers who uncritically accept algorithm outputs may miss errors, overlook relevant studies, or adopt conclusions that the underlying data do not support. This is particularly concerning in systematic reviews, where the consequences of missing relevant studies or accepting biased evidence can propagate through clinical guidelines and affect patient care. Understanding automation bias is essential for developing appropriate oversight procedures that capture the efficiency benefits of automation while maintaining the quality and integrity of research.

Automation bias is commonly explained by two mechanisms. The first is complacency, in which the user fails to notice errors or anomalies in the automated output because they assume the system is correct. The second is anchoring, in which the user's independent judgment is biased toward the automated recommendation, making them less likely to reach a different conclusion even when the evidence supports one. Together these produce two kinds of error: omission errors, where a problem goes unnoticed because the system did not flag it, and commission errors, where the user follows a recommendation that is wrong.

In systematic review screening, automation bias can manifest when reviewers use machine learning tools to prioritize records. If the algorithm assigns a low relevance score to a study that is actually eligible, a reviewer who is influenced by the score may be more likely to exclude it than they would have been without the score. Conversely, a high relevance score may lead reviewers to include borderline studies that they might otherwise have excluded on closer examination.

In data analysis, automation bias can occur when researchers accept the default output of statistical software without verifying that the assumptions of the test are met, the variables are correctly specified, or the results make substantive sense. The ease of running analyses with a few clicks can create a false sense of precision that discourages critical evaluation of the output.

Several factors increase susceptibility to automation bias: high workload (when reviewers are fatigued, they are more likely to defer to the algorithm), high trust in the technology (users who believe the system is highly accurate are less likely to question it), and low confidence in their own judgment (users who feel uncertain are more likely to rely on the automated recommendation), although expertise alone does not protect, since experienced users also show the effect.

Awareness of automation bias is relevant whenever automated tools are integrated into research workflows. Researchers should develop and apply strategies to mitigate it, such as requiring independent human judgment before automated scores are revealed, establishing protocols for manually verifying a random sample of automated decisions, and training team members to recognize the cognitive mechanisms through which automation bias operates. These safeguards are particularly important in high stakes contexts such as systematic review screening, clinical data analysis, and regulatory submissions.

A common error is assuming that awareness of automation bias is sufficient to prevent it. Research on cognitive biases shows that knowing about a bias does not reliably remove susceptibility to it; structural safeguards such as dual review, blind verification, and mandatory override protocols are needed as well.

Another mistake is attributing errors entirely to the automated system when the real problem is inadequate human oversight. If a machine learning screening tool misses a relevant study, the failure may lie not in the algorithm but in the review protocol that did not include sufficient manual checks. Researchers should also avoid the opposite extreme of rejecting automated tools entirely because of automation bias concerns; the goal is to integrate automation with appropriate oversight, not to abandon it. Finally, failing to document how automation was used, what checks were applied, and how disagreements between human and automated judgments were resolved undermines the reproducibility and transparency of the research.

A systematic review team uses an active learning algorithm to assist with title and abstract screening. After training the model on 400 manually screened records, the algorithm assigns relevance scores to the remaining 8,600 records. To mitigate automation bias, the team implements a protocol in which each reviewer screens records without seeing the algorithm's score for the first pass. A second reviewer then screens the same records with the score visible. When the two assessments disagree, a third reviewer adjudicates. This dual review process identifies 14 records that the algorithm scored as low relevance but that the reviewers judged worth retrieving at title and abstract screening; seven of these proved eligible after full text assessment.