A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants
PLoS ONEPublished 14 December 2010Open access
Lutz Bornmann, Rüdiger Mutz, Hans‐Dieter Daniel
Citations221
SJR quartileQ1
SJR score0.80
SNIP1.07
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
According to the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews, the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
Abstract
Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
Keywords
Decision Sciences
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Controlled Clinical TrialsMeta-analysis in clinical trials
39,316 Citations1986Rebecca DerSimonian, Nan M. Laird
This paper examines eight published reviews each reporting results from several related trials in order to evaluate the efficacy of a certain treatment for a specified medical condition and suggests a simple noniterative procedure for characterizing the distribution of treatment effects in a series of studies.
Experimental and quasi-experimental designs for generalized causal inference
13,404 Citations2010William R. Shadish, Thomas D. Cook +1 more
Cochrane Database of Systematic ReviewsPharmacological interventions for somatoform disorders in adults
11,355 Citations2014Maria Kleinstäuber, Michael Witthöft +4 more
A systematic review and meta-analysis of placebo-controlled studies examined the efficacy and tolerability of different types of antidepressants, the combination of an antidepressant and an antipsychotic, antipsychotics alone, or natural products in adults with somatoform disorders in adults to improve optimal treatment decisions.
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
5,538 Citations2015Frank L. Schmidt, John E. Hunter
Organizational Research MethodsAnswers to 20 Questions About Interrater Reliability and Interrater Agreement
3,506 Citations2007James M. LeBreton, Jenell L. Senter
Psychological MethodsFixed- and random-effects models in meta-analysis.
2,838 Citations1998Larry V. Hedges, Jack L. Vevea
Contemporary Clinical TrialsRandom-effects model for meta-analysis of clinical trials: An update
2,344 Citations2006Rebecca DerSimonian, Raghu N. Kacker
It is shown that the leading methods for estimating the inter-study variance are special cases of a general method-of-moments estimate of the inter"-study variance" and suggested two new two-step methods.
Statistics in MedicineAdvanced methods in meta‐analysis: multivariate approach and meta‐regression
1,577 Citations2002Hans C. van Houwelingen, Lidia R. Arends +1 more
This tutorial on advanced statistical methods for meta-analysis can be seen as a sequel to the recent Tutorial in Biostatistics on meta- analysis by Normand, which focused on elementary methods.
JAMAThe Existence of Publication Bias and Risk Factors for Its Occurrence
1,345 Citations1990Kay Dickersin
Cochrane Database of Systematic ReviewsPublication bias in clinical trials due to statistical significance or direction of trial results
887 Citations2009Sally Hopewell, Kirsty Loudon +3 more
To assess the extent to which publication of a cohort of clinical trials is influenced by the statistical significance, perceived importance, or direction of their results, five studies were included and found no significant association between either funding mechanism, investigator rank, or sex and publication.
International Journal of Selection and AssessmentFixed Effects vs. Random Effects Meta‐Analysis Models: Implications for Cumulative Research Knowledge
835 Citations2000John E. Hunter, Frank L. Schmidt
Journal of the Royal Society of MedicinePeer review: a flawed process at the heart of science and journals
815 Citations2006Judith Smith
Real science what it is, and what it means
710 Citations2000John Ziman
Psychological MethodsMeta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods.
708 Citations2001Andy P. Field
Two Monte Carlo simulations are presented that compare the efficacy of the Hedges and colleagues, Rosenthal-Rubin, and Hunter-Schmidt methods for combining correlation coefficients for cases in which population effect sizes were both fixed and variable.
Methods of meta-analysis correcting error and bias in research findings
565 Citations1990Hunter, John Edward 1939-2002, Schmidt, Frank L. 1944-
Behavioral and Brain SciencesThe reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation
502 Citations1991Domenic V. Cicchetti
International Journal of Clinical PracticeUnderstanding heterogeneity in meta-analysis: the role of meta-regression
463 Citations2009William L. Baker, C Michael White +4 more
This work states that meta‐regression has grown in popularity in recent years, paralleling the increasing numbers of systematic reviews and meta‐analysis published in the biomedical literature, but many clinicians and decision‐makers may be unfamiliar with the underlying principles and assumptions made within meta-regression leading to incorrect interpretation of their results.
Educational and Psychological MeasurementReliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies
446 Citations1998Tammi Vacha‐Haase
Journal of InformetricsGender differences in grant peer review: A meta-analysis
346 Citations2007Lutz Bornmann, Ruediger Mutz +1 more
A meta-analysis of 21 studies providing evidence of robust gender differences in grant award procedures shows that among grant applicants men have statistically significant greater odds of receiving grants than women by about 7%.
Educational and Psychological MeasurementPsychometrics is Datametrics: the Test is not Reliable
342 Citations2000Bruce Thompson, Tammi Vacha‐Haase
American PsychologistImproving the peer-review process for grant applications: Reliability, validity, bias, and generalizability.
293 Citations2008Herbert W. Marsh, Upali W Jayasinghe +1 more
Journal of Clinical EpidemiologyAgreement among reviewers of review articles
283 Citations1991Andrew D Oxman, Gordon Guyatt +5 more
Assessment of the consistency of an index of the scientific quality of research overviews found it was possible to achieve reasonable to excellent agreement for all of the items in the index, including the overall assessment of scientific quality.
BrainReproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?
250 Citations2000Peter M. Rothwell
Although recommendations made by reviewers have considerable influence on the fate of both papers submitted to journals and abstracts submitted to conferences, agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.
DigitalGeorgetown (Georgetown University Library)Editorial peer review : its strengths and weaknesses
233 Citations2001Ann C. Weller
Educational and Psychological MeasurementMultilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures
207 Citations2003Wim Van Den Noortgate, Patrick Onghena
The results of the simulation study suggest that the maximum likelihood multilevel approach is in general superior to the fixed-effects approaches, unless only a small number of studies is available.
Social Studies of ScienceThe Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome
206 Citations2001Liv Langfeldt
Science CommunicationPeer Review for Journals as it Stands Today—Part 1
200 Citations1998Juan Miguel Campanario
It is difficult to imagine how science could advance without such a key quality control mechanism as peer review, the author concludes.
Information Today eBooksThe web of knowledge: a festschrift in honor of Eugene Garfield
182 Citations2000Eugene Garfield, Blaise Cronin +1 more
Review of Educational ResearchGender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches
179 Citations2009Herbert W. Marsh, Lutz Bornmann +3 more
The Economics of non-market goods and resourcesContrasting conventional with multi-level modeling approaches to meta-analysis: Expectation consistency in UK woodland recreation values
173 Citations2007Ian J. Bateman, Andy Jones
JAMAMaking Reviewers Visible
166 Citations2002Fiona Godlee
I look forward to a time when open commentary and review replace the current, flawed system of closed prepublication peer review and its false reassurances about the reliability of what is published.
Land EconomicsContrasting Conventional with Multi-Level Modeling Approaches to Meta-Analysis: Expectation Consistency in U.K. Woodland Recreation Values
161 Citations2003Ian J. Bateman, Andy Jones
International Journal of EpidemiologyWhere now for meta-analysis?
159 Citations2002Matthias Egger, Shah Ebrahim +1 more
This issue of the International Journal of Epidemiology contains several papers that address methodological issues in metaanalytic research, a review article on where the authors stand with systematic reviews in observational epidemiology 10 and three meta-analyses of observational studies.
BMC Health Services ResearchInter-rater agreement in the scoring of abstracts submitted to a primary care research conference
147 Citations2002Alan Montgomery, Anna R. Graham +2 more
The findings suggest that inclusion of subjective components in a review checklist may result in greater disagreement between reviewers, however in terms of overall quality scores, abstracts accepted for the meeting were rated significantly higher than those that were rejected.
PubMedA statistical analysis of reviewer agreement and bias in evaluating medical abstracts.
144 Citations1976Domenic V. Cicchetti, Harold O. Conn
Of the 77 abstracts rated for all parameters by all observers, interobserver agreement ranged between 81 and 88%.
Science CommunicationPeer Review for Journals as it Stands Today—Part 2
134 Citations1998Juan Miguel Campanario
Academy of Management JournalTHE REVIEW PROCESS AND THE FATES OF MANUSCRIPTS SUBMITTED TO AMJ.
134 Citations1995Janice M. Beyer, Roland G. Chanove +1 more
Journal of the Royal Statistical Society Series C (Applied Statistics)Meta-Analysis Using Multilevel Models with an Application to the Study of Class Size Effects
133 Citations2000Harvey Goldstein, Min Yang +3 more
ACM SIGMOD RecordSingle- versus double-blind reviewing
123 Citations2006Richard T. Snodgrass
The peer review process is generally acknowledged as central to the advancement of scholarly knowledge and it is also vital to the development of individual careers.
JAMAReliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
120 Citations1998Michael L. Callaham
Subjective editor ratings of individual reviewers were moderately reliable and correlated with reviewer ability to report manuscript flaws, but these variables were poorly correlated with editors' ratings of review quality or the reviewer's ability to detect flaws in a fictitious manuscript.
BMC Medical Research MethodologyDevelopment and evaluation of a quality score for abstracts
119 Citations2003Antje Timmer, Lloyd R. Sutherland +1 more
Harvard Educational ReviewEvaluating the Effect of Coaching on SAT Scores: A Meta-Analysis
116 Citations1983Rebecca DerSimonian, Nan M. Laird
Medical Entomology and ZoologyGuardians of Science: Fairness and Reliability of Peer Review
110 Citations1994Hans‐Dieter Daniel, William E. Russey
JAMAMasking Author Identity in Peer Review
101 Citations1998Mildred K. Cho, Amy C. Justice +6 more
Masking success appears unrelated to a journal policy of masking, but is associated with reviewers' research experience and could be affected by other characteristics.
Science Technology & Human ValuesThe Shift in Academic Quality Control
99 Citations2006Sven Hemlin, Søren Barlebo Rasmussen
ScientometricsAssessing precision in the manuscript review process: A little better than a dice roll
98 Citations1988Duncan Lindsey
The findings indicate that highly selective decision-making with imprecise reviewers results in outcomes that are only slightly better than chance.
Educational and Psychological MeasurementUsing Mixed-Effects Models In Reliability Generalization Studies
78 Citations2003S. Natasha Beretvas, Dena A. Pastor
Angewandte Chemie International EditionThe Effectiveness of the Peer Review Process: Inter‐Referee Agreement and Predictive Validity of Manuscript Refereeing at <i>Angewandte Chemie</i>
69 Citations2008Lutz Bornmann, Hans‐Dieter Daniel
The quality of peer review at Angewandte Chemie is examined according to the criteria of reliability (agreement among referees) and predictive validity, using an optimized study design and modern methods of statistical data analysis.
Social Science ResearchA new approach to referees' assessments of manuscripts
66 Citations1990Lowell L. Hargens, Jerald R. Herting
Journal of General Internal MedicineHow reliable is peer review of scientific abstracts?
63 Citations1993Haya R. Rubin, Donald A. Redelmeier +2 more
Global ratings were more reproducible than summary scores, and future meeting organizers may wish to rank abstracts using global ratings, and to experiment with structured review criteria and other ways to improve raters’ agreement.
Personality and Social Psychology BulletinThe Review Process at PSPB: Correlates of Interreviewer Agreement and Manuscript Acceptance
62 Citations1999Richard E. Petty, Monique A. Fleming +1 more
Evaluation & the Health ProfessionsThe Reliability of Peer Assessments
62 Citations1994Ronald L. Goldman
A meta-analysis of studies examining the interrater reliability of the standard practice of peer assessments of quality of care suggests that the interloper reliability of peer assessment is quite limited and needs improvement.
Psychological ScienceAn Examination of Sources of Peer-Review Bias
57 Citations2006Jessica L. Blackburn, Milton D. Hakel
Peer-review ratings of 1,983 posters submitted for three annual conferences of a professional society were examined for evidence of bias and differences in reliability and acceptance decisions were examined.
Behavioral and Brain SciencesReliability, fairness, objectivity and other inappropriate goals in peer review
54 Citations1991John C. Bailar
American Journal of PsychiatryAn intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry
53 Citations1993Joseph M. Strayhorn, John Francis McDermott +1 more
The data confirm that the reliability of ratings of scientific articles may be improved by increasing the number of rating scale points, eliciting ratings of separate, concrete items rather than a global judgment, using training manuals, and averaging the scores of multiple reviewers.
Developmental ReviewInterrater agreement for reviews for Developmental Review
47 Citations1983Grover J. Whitehurst
JAMADo Readers and Peer Reviewers Agree on Manuscript Quality?
43 Citations1994Amy C. Justice
Readers, like most peer reviewers, are generally satisfied with the quality of manuscripts but would like research articles to be more relevant to their clinical practice.
The British Journal of PsychiatryPeer review and editorial decision-making
42 Citations1998Louise M. Howard, Greg Wilkinson
The editor's decision-making process at the British Journal of Psychiatry (BJP) is described and analyses the association between reviewers' assessments and editorial decisions, and there is less agreement on what is suitable.
ScientometricsA new reader trial approach to peer review in funding research grants: An Australian experiment
38 Citations2006Upali W Jayasinghe, Herbert W. Marsh +1 more
In comparison to the traditional peer review approach, the new reader system is substantially more reliable, timely, and cost efficient - and applicable to many peer review situations.
Social Work ResearchAgreeing to disagree: A study of the reliability of manuscript reviews
37 Citations1997S. A. Kirk, Todd Franke
Australian PsychologistPeer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid
37 Citations2007Herbert W. Marsh, Nigel W. Bond +1 more
Indian PediatricsWhat is submitted and what gets accepted in Indian Pediatrics: Analysis of submissions, review process, decision making, and criteria for rejection
34 Citations2013Piyush Gupta, Geetinder Kaur +3 more
The top reasons for rejection were ‘absence of a message’, ‘lack of originality’.
Journal of Information ScienceThe Reliability of Peer Reviews of Papers on Information Systems
30 Citations2004Michael Wood, Martyn Roberts +1 more
The reliability of the double-blind peer review systems used for submissions to the 2001 and 2002 UKAIS conferences is analyzed, and a model for assessing the impact of low levels of reliability is suggested.
BMJGuardians of Science: Fairness and Reliability of Peer Review
28 Citations1994Sara Fletcher
H-D Daniel, VCH, (pounds sterling)25, pp 118, ISBN 3-527-29041-9 Early this century an eccentric Boston surgeon suggested that surgeons should follow up their patients to determine how often surgery was successful and to analyse failures for future improvement.
Plastic & Reconstructive SurgeryReliability of a Structured Method of Selecting Abstracts for a Plastic Surgical Scientific Meeting
27 Citations2003Lydia P. E. van der Steen, J. Joris Hage +2 more
The development and prospective evaluation of a structured method of selection applied to 220 abstracts submitted for the 2000 Annual Meeting of the European Association of Plastic Surgeons and indicated an acceptable degree of reliability of the average reviewers’ ratings for all categories.
Angewandte Chemie International Edition in EnglishAn Evaluation of the Peer Review Process at <i>Angewandte Chemie</i>
21 Citations1993Hans‐Dieter Daniel
Archives of Pediatrics and Adolescent MedicineImproving Participation and Interrater Agreement in Scoring Ambulatory Pediatric Association Abstracts
20 Citations1996Kathi J. Kemper
The number and expertise of those rating abstracts increased from 1990 to 1995, however, interrater agreement did not change and remained low, and further efforts are needed to improve the interRater agreement.
Clinical Orthopaedics and Related ResearchInterrater Reliability in Grading Abstracts for the Orthopaedic Trauma Association
18 Citations2004Mohit Bhandari, David C. Templeman +1 more
Examination of the consistency of reviewers in grading abstracts submitted for podium presentations at the 2001 and 2002 Annual Meetings of the Orthopaedic Trauma Association finds that agreement among unblinded reviewers of the 67 and 73 podium presentations during the2001 and 2002 meetings did not improve interreviewer agreement.
PubMedPeer review in a small and a big medical journal: case study of the Croatian Medical Journal and the Lancet.
18 Citations2002Ana Marušić, Ivan Krešimir Lukić +4 more
Reviewers of big journals, such as The Lancet, are stricter in their recommendations reviewers of a smaller journal,such as the CMJ, which relies on reviewers to identify methodologically superior studies, whereas CMJ editors look for the novelty of information in a manuscript.
BMJFifth international congress on peer review and biomedical publication
14 Citations2003Drummond Rennie, Annette Flanagin +2 more
It is shown that since the announcement of the first peer review congress in 1986, the numbers of abstracts submitted to successive congresses have increased from 50 to 110, 160, and 180, respectively, and similar increases in the number of Medline citations on peer review have been noted.
Research Quarterly for Exercise and SportInterrater Reliability of 1987–1991<i>Research Quarterly for Exercise and Sport</i>Reviews
8 Citations1992James R. Morrow, Molly S. Bray +2 more
Anesthesia & AnalgesiaPeer Review Interrater Concordance of Scientific Abstracts: A Study of Anesthesiology Subspecialty and Component Societies
6 Citations2006Ira Todd Cohen, Kantilal M. Patel
It is hypothesize that having clearer evaluation criteria, scoring systems with interval scales, and assessment based on quality can strengthen the peer review process.
Mayo Clinic ProceedingsHow Masked Is the “Masked Peer Review” of Abstracts Submitted to International Medical Conferences?
6 Citations2006Matthew E. Falagas, George M. Zouglakis +1 more
Journal of Economic PsychologyEditorial Comment: Agreement between reviewers of Journal of Economic Psychology submissions
6 Citations2005Simon Kemp
PubMedA worldwide assessment of medical journal editors' practices and needs--results of a survey by the World Association of Medical Editors.
6 Citations1999Cynthia D. Good, Stephen T. Parente +2 more
There was strong interest among respondents in a global organisation for medical editors, and half of the responding editors reported having access to the Internet, making participation in a worldwide computer network of editors feasible.
AnthrozoösThe reliability of peer review in anthrozoology
4 Citations2005Harold A. Herzog, Anthony L. Podberscek +1 more
South African Journal of PsychologyThe Reliability of Manuscript Evaluation for the <i>South African Journal of Psychology</i>
3 Citations1993C. Plug
Journal of Education in Perioperative MedicinePeer Review Interrater Reliability of Scientific Abstracts: A Study of an Anesthesia Subspecialty Society
2 Citations2005Ira Todd Cohen, Kantilal M. Patel
A low level of interrater reliability was found among reviewers of abstracts submitted for presentation at an anesthesiology subspecialty society's annual meeting, similar to that found for abstract scoring by other medical subspecialties.
…