Longitudinal Trends in the Performance of Scientific Peer Reviewers
Annals of Emergency MedicinePublished 13 November 2010Open access
Michael L. Callaham, Charles E. McCulloch
Citations64
SJR quartileQ1
SJR score0.84
SNIP1.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is demonstrated that most journal peer reviewers received lower quality scores for article assessment over the years, due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations.
Abstract
This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.
Keywords
Computer ScienceDecision Sciences
Journal of Personality and Social PsychologyUnskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments.
6,908 Citations1999Justin Kruger, David Dunning
Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability.
Wiley series in probability and statisticsApplied Longitudinal Analysis
4,568 Citations2011Garrett M. Fitzmaurice, Nan M. Laird +1 more
Cambridge University Press eBooksThe Cambridge Handbook of Expertise and Expert Performance
3,151 Citations2006K. Anders Ericsson, K. Anders Ericsson +20 more
Introduction and perspective -- An introduction to Cambridge handbook of expertise and expert performance : its development, organization, and content is presented.
Academic MedicineDeliberate Practice and the Acquisition and Maintenance of Expert Performance in Medicine and Related Domains
2,945 Citations2004K. Anders Ericsson
This article proposes an alternative framework to account for individual differences in attained professional development, as well as many aspects of age-related decline, based on the assumption that acquisition of expert performance requires engagement in deliberate practice and that continued deliberate practice is necessary for maintenance of many types of professional performance.
Annual Review of PsychologyEXPERT AND EXCEPTIONAL PERFORMANCE: Evidence of Maximal Adaptation to Task Constraints
2,123 Citations1996K. Anders Ericsson, Andreas Lehmann
Many of the mechanisms of superior expert performance serve the dual purpose of mediating experts' current performance and of allowing continued improvement of this performance in response to informative feedback during practice activities.
Annals of Internal MedicineSystematic Review: The Relationship between Clinical Experience and Quality of Health Care
1,474 Citations2005Niteesh K. Choudhry, Robert H. Fletcher +1 more
The robustness of the relationship between clinical experience and quality of care is assessed by systematically reviewing empirical studies to assess the effects of experience on the quality of medical care.
American Economic ReviewResearch productivity over the life cycle: evidence for academic scientists
790 Citations1991Sharon G. Levin, Paula E. Stephan
Advances in Health Sciences EducationHow Can I Know What I Don't Know? Poor Self Assessment in a Well-Defined Domain
300 Citations2004Kevin W. Eva, John Cunnington +3 more
Medical EducationAn expert‐performance perspective of research on medical expertise: the study of clinical performance
276 Citations2007K. Anders Ericsson
This work describes the failure to identify superior performance by peer‐nominated diagnosticians using high‐ and low‐fidelity simulations of the everyday practice of doctors.
BMJEffects of training on quality of peer review: randomised controlled trial
268 Citations2004Sara Schroter, Nick Black +4 more
Training had no impact on the time taken to review the papers but was associated with an increased likelihood of recommending rejection and the value of longer interventions needs to be assessed.
JAMAWhat Makes a Good Reviewer and a Good Review for a General Medical Journal?
266 Citations1998Nick Black
The characteristics of reviewers studied did not identify those who performed high-quality reviews, and reviewers might be advised that spending longer than 3 hours on a review on average did not appear to increase review quality as rated by editors and authors.
Statistics in MedicineAn introduction to hierarchical linear modelling
248 Citations1999Lisa Sullivan, Kimberly A. Dukes +1 more
This tutorial provides an introduction to the hierarchical linear models technique in general terms, and then specifies model notation and assumptions in detail, and elaborate on model interpretation and provide guidelines for model checking.
Psychology Press eBooksComplex Problem Solving
248 Citations2014Robert J. Sternberg, Peter A. Frensch
Journal of the Royal Society of MedicineWhat errors do peer reviewers detect, and does training improve their ability to detect them?
227 Citations2008Sara Schroter, Nick Black +4 more
IntelligenceAre intelligence tests measurement invariant over time? Investigating the nature of the Flynn effect
206 Citations2004Jelte M. Wicherts, Conor V. Dolan +5 more
Academic MedicineThe Aging Physician
189 Citations2002Kevin W. Eva
This paper systematically examines the small amount of evidence available in medical education that highlights the issue of aging and attempts to reconcile contradictory findings by drawing on the much larger psychological literature on pre-senile aging.
Journal of Clinical EpidemiologyDevelopment of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts
174 Citations1999Susan van Rooyen, Nick Black +1 more
A Review Quality Instrument that assesses the extent to which a reviewer has commented on five aspects of a manuscript and on two aspects of the review (constructiveness and substantiation of comments) was devised and tested and recommended for use in the study of peer review.
Journal of General Internal MedicineThe characteristics of peer reviewers who produce good-quality reviews
151 Citations1993Arthur T. Evans, Robert Mcnutt +2 more
Good peer reviewers for this journal tended to be young, from strong academic institutions, well known to the editors, and blinded to the identity of the manuscript’s authors.
Annals of Emergency MedicineWho Reviews the Reviewers? Feasibility of Using a Fictitious Manuscript to Evaluate Peer Reviewer Performance
140 Citations1998William G. Baxt, Joseph F. Waeckerle +2 more
The use of a preconceived manuscript into which purposeful errors are placed may be a viable approach to evaluate reviewer performance and peer reviewers in this study failed to identify two thirds of the major errors.
PLoS MedicineThe Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality
139 Citations2007Michael L. Callaham, John Anthony Tercier
It is confirmed that there are no easily identifiable types of formal training or experience that predict reviewer performance, and it is imperative that all but the smallest journals implement routine review ratings systems to routinely monitor the quality of their reviews.
JAMAMultiple Blinded Reviews of the Same Two Manuscripts
131 Citations1994Magne Nylenna
Experienced and young referees gave a stricter assessment of the manuscripts than their less experienced and older colleagues, and an English version seemed to be accepted more easily than a national-language version of the same manuscript.
Academic MedicineCertification and Specialization
125 Citations2000John J. Norcini, Harry R. Kimball +1 more
Less patient mortality was associated with treatment by physicians who were cardiologists, cared for larger numbers of AMI patients, were closer to their graduation from medical school, and were certified.
JAMAReliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
120 Citations1998Michael L. Callaham
Subjective editor ratings of individual reviewers were moderately reliable and correlated with reviewer ability to report manuscript flaws, but these variables were poorly correlated with editors' ratings of review quality or the reviewer's ability to detect flaws in a fictitious manuscript.
Journal of Research in PersonalityWhy productivity fades with age: The crime–genius connection
102 Citations2003Satoshi Kanazawa
ScientometricsAge and the Nobel prize revisited
92 Citations1993Paula E. Stephan, Sharon G. Levin
The results suggest that although it does not require extraordinary youth to do prizewinning work, the odds decrease markedly in mid-life and fall off precipitously after age 50, particularly in chemistry and physics.
Annals of Emergency MedicineEffect of Attendance at a Training Session on Peer Reviewer Quality and Performance
79 Citations1998Michael L. Callaham, Robert L. Wears +1 more
In a self-selected group of experienced reviewers who attended a 4-hour workshop on peer review, no effect could be identified in subsequent performance as measured by editors' quality ratings or reviewer performance statistics.
JAMAEffect of Written Feedback by Editors on Quality of Reviews
72 Citations2002Michael L. Callaham, Robert Knopp +1 more
In study 1, minimal feedback from editors on review quality had no effect on subsequent performance of poor-quality reviewers, and the trend was toward a negative effect.
Journal of Continuing Education in the Health ProfessionsThe difficulty with experience: Does practice increase susceptibility to premature closure?
70 Citations2006Kevin W. Eva, John Cunnington
Physicians with greater experience appear to weigh their first impressions more heavily than those with less experience, and Educators should design instructional activities that account for experience‐specific cognitive tendencies.
New England Journal of MedicineReviewer Status and Review Quality
68 Citations1985Thomas P. Stossel
Communication in biomedical science, as currently structured, assumes the voluntary participation of all researchers in the peer review of articles submitted for publication.
American Journal of RoentgenologyReviewing the Reviewers: Comparison of Review Quality and Reviewer Characteristics at the <i>American Journal of Roentgenology</i>
64 Citations2005Mark A. Kliewer, K S Freed +3 more
The highest-rated AJR reviewers tended to be young and from academic institutions, and the quality of peer review did not correlate with the sex, academic rank, or subspecialty of the reviewer.
BMC Medical Research MethodologyContent and communication: How can peer review provide helpful feedback about the writing?
54 Citations2008Karen Shashok
Annals of Emergency MedicineEffect of structured workshop training on subsequent performance of journal peer reviewers
52 Citations2002Michael L. Callaham, David L. Schriger
Obstetrics and GynecologyQuality Assessment of Reviewers’ Reports Using a Simple Instrument
30 Citations2006Annemieke P. Landkroon, Anne M. Euser +3 more
This 5-point scale proved to be a simple, reliable, and valid instrument enabling editors to assess the quality of reviews and a significant correlation was found between mean editorial quality assessment and the quality as determined by authors.
Annals of Emergency MedicineImplementation of a Journal Peer Reviewer Stratification System Based on Quality and Reliability
24 Citations2010Steven M. Green, Michael L. Callaham
Implementation of a tiered system stratifying journal peer reviewers by quality and reliability was readily accomplished by Annals and has appeared to improve the efficiency of the peer review.
New England Journal of MedicineWho Reviews the Reviewers?
11 Citations1980Mark K. Adler, Curtland C. Brown
In the New England region, medical information submitted to a Medicare fiscal intermediary is reviewed by a registered nurse who makes the initial coverage determination, following the guidelines in Medicare's Intermediary Manual.
eScholarship (California Digital Library)A normative model of peer review: qualitative assessment of manuscript reviewers’ attitudes towards peer review
7 Citations2007John Anthony Tercier, Michael L. Callaham
The tensions found in the peer review process, sometimes seen as barriers to its effective operation, are less defects in the process than definitive of the concept of peer review itself and thus necessary to its operation.