Reviewer bias: a blinded experimental study.

Evaluating interrater consistency in reviewing a single manuscript and determining whether a referee's likely predisposition influences his or her attitude toward this manuscript suggest significant impact of reviewer bias on referee's judgment and imply that the peer review system in its present form has room for improvements in fairness and consistency.

Abstract

Peer review is the ordinary method for judging the acceptability of scientific papers for publication. Yet there are few prospective data defining the accuracy or reproducibility of this method. This study was aimed at evaluating interrater consistency in reviewing a single manuscript and at determining whether a referee's likely predisposition (inferred from his or her own published papers in the field) influences his or her attitude toward this manuscript. A computerized search identified 33 first authors of research papers about transcutaneous electrical nerve stimulation. They were asked to review part of a fictitious scientific paper on the topic; they were not made aware of being tested. Reviewers' ratings were recorded by multiple-choice answers in a structured questionnaire. The answers revealed poor interrater reliability. Furthermore, referees were clearly influenced by their own preconceptions and judged according to their own published experience in that particular medical subject. That is, referees who would be expected to agree with the paper's findings tended to judge is less harshly than did referees who would be expected to disagree. These results suggest significant impact of reviewer bias on referee's judgment and imply that the peer review system in its present form has room for improvements in fairness and consistency.