Reliability, fairness, objectivity and other inappropriate goals in peer review
Behavioral and Brain SciencesPublished 1 March 1991
John C. Bailar
Citations54
SJR quartileQ2
SJR score0.55
SNIP0.43
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
Keywords
Social SciencesHealth ProfessionsEconomics, Econometrics and Finance
Computers Environment and Urban SystemsStatistical power analysis for the behavioral sciences
65,645 Citations1990
Physics Today<i>The Structure of Scientific Revolutions</i>
35,519 Citations1963Thomas Kühn, Richard Schlegel
Psychological BulletinThe file drawer problem and tolerance for null results.
8,595 Citations1979Robert Rosenthal
Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
American Educational Research JournalStatistical Theories of Mental Test Scores
8,138 Citations1969William W. Rozeboom, Frederic M. Lord +2 more
Cognitive Therapy and the Emotional Disorders
6,314 Citations1975Aaron T. Beck
Journal of the Royal Statistical Society Series D (The Statistician)Meta-Analytic Procedures for Social Research.
4,283 Citations1994Andrew Metcalfe, R. Rosenthal
Digital Access to Scholarship at Harvard (DASH) (Harvard University)Essentials of Behavioral Research: Methods and Data Analysis
3,964 Citations1984Robert Rosenthal, Ralph L. Rosnow
This chapter discusses the nature of Behavioral Research, the development and testing of research ideas, and the selection of subjects and Stimuli for experiments.
Journal of Personality and Social PsychologyBystander intervention in emergencies: Diffusion of responsibility.
2,963 Citations1968John M. Darley, Bibb Latané
This experiment suggests that the explanation for bystander inaction in real-life emergencies may lie more in the bystander's response to other observers than in his indifference to the victim.
Journal of Clinical EpidemiologyHigh agreement but low Kappa: I. the problems of two paradoxes
2,886 Citations1990Alvan R. Feinstein, Domenic V. Cicchetti
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance.
ScienceCitation Analysis as a Tool in Journal Evaluation
2,760 Citations1972Eugene Garfield
In 1971, the Institute for Scientfic Information decided to undertake a systematic analysis of journal citation patterns across the whole of science and technology.
PubMedDeveloping criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior.
2,278 Citations1981Domenic V. Cicchetti, Sara A. Sparrow
A set of criteria based upon biostatistical considerations for determining the interrater reliability of specific adaptive behavior items in a given setting was presented and guidelines for differentiating type of adaptive behavior that are statistically reliable from those that are reliable in a clinical or practical sense were delineated.
Quarterly Journal of Experimental PsychologyOn the Failure to Eliminate Hypotheses in a Conceptual Task
2,087 Citations1960P. C. Wason
The results showed that those subjects, who reached two or more incorrect conclusions, were unable, or unwilling to test their hypotheses, and the implications are discussed in relation to scientific thinking.
DigitalGeorgetown (Georgetown University Library)Science as a Process: An Evolutionary Account of the Social and Conceptual Development of Science
2,061 Citations1988David L. Hull
Psychological ReviewConfirmation, disconfirmation, and information in hypothesis testing.
2,055 Citations1987Joshua Klayman, Young-Won Ha
Journal of Clinical EpidemiologyHigh agreement but low kappa: II. Resolving the paradoxes
1,836 Citations1990Domenic V. Cicchetti, Alvan R. Feinstein
An omnibus index offers a single summary expression for a fourfold table of binary concordance among two observers and the paradoxes of kappa are desirable since they appropriately "penalize" inequalities in ppos and pneg.
Psychological BulletinLarge sample standard errors of kappa and weighted kappa.
1,504 Citations1969Joseph L. Fleiss, Jacob Cohen +1 more
JAMAThe Existence of Publication Bias and Risk Factors for Its Occurrence
1,345 Citations1990Kay Dickersin
The Journal of Nervous and Mental DiseaseON THE METHODS AND THEORY OF RELIABILITY
1,317 Citations1976John J. Bartko, William T. Carpenter
This paper reviews the most frequently used and misused reliability measures appearing in the mental health literature and suggests some suitable reliability measures to be used.
The Philosophical ReviewExplaining Science: A Cognitive Approach.
1,301 Citations1991Jeffrey S. Poland, Barbara Von Eckardt +1 more
Journal of the American Statistical AssociationContrast Analysis: Focused Comparisons in the Analysis of Variance.
1,277 Citations1987Joel R. Levin, Robert Rosenthal +1 more
Behavioral and Brain SciencesPeer-review practices of psychological journals: The fate of published articles, submitted again
1,080 Citations1982Douglas Peters, Stephen J. Ceci
Psychological BulletinConsequences of prejudice against the null hypothesis.
1,054 Citations1975Anthony G. Greenwald
Cognitive Therapy and ResearchPublication prejudices: An experimental study of confirmatory bias in the peer review system
1,030 Citations1977M. J. Mahoney
Journal of the American Statistical AssociationPublication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
936 Citations1959T. D. Sterling
Journal of Educational PsychologyA simple, general purpose display of magnitude of experimental effect.
932 Citations1982Robert Rosenthal, Donald B. Rubin
NatureA History of Psychology in Autobiography
864 Citations1937Gardner Lindzey, Murchison, Carl Allanmore, 1887-
BMJResults of case-control study of leukaemia and lymphoma among young people near Sellafield nuclear plant in West Cumbria.
783 Citations1990Martin J. Gardner, M. Snee +4 more
The raised incidence of leukaemia, particularly, and non-Hodgkin's lymphoma among children near Sellafield was associated with paternal employment and recorded external dose of whole body penetrating radiation during work at the plant before conception, suggesting an effect of ionising radiation on fathers that may be leukaemogenic in their offspring.
Journal of the American Statistical AssociationMeasures of Association for Cross Classifications*
674 Citations1954Leo A. Goodman, William Kruskal
New England Journal of MedicineThe Reliability of Clinical Methods, Data and Judgments
652 Citations1975Martín Keller
ScienceChance and Consensus in Peer Review
624 Citations1981Stephen Cole, Jonathan R. Cole +1 more
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance.
The British Journal of PsychiatryA Re-analysis of the Reliability of Psychiatric Diagnosis
615 Citations1974Robert L. Spitzer, Joseph L. Fleiss
Classification systems such as diagnosis have two primary properties, reliability and validity, which refers to the consistency with which subjects are classified and the utility of the system for its various purposes.
JAMAUnderreporting Research Is Scientific Misconduct
556 Citations1990lain Chalmers
An extended use of prospective registration of trials at inception, as well as benefiting clinical research in other ways, could help people to play their respective roles in reducing underreporting of clinical trials.
Psychological ReviewUnder what conditions does theory obstruct research progress?
531 Citations1986Anthony G. Greenwald, Anthony R. Pratkanis +2 more
Researchers display confirmation bias when they persevere by revising procedures until obtaining a theory-predicted result, which produces findings that are overgeneralized in avoidable ways, and this in turn hinders successful applications.
The Journal of Higher EducationScience and Values: The Aims of Science and Their Role in Scientific Debate
523 Citations1987Richard Whitley, Larry Laudan
Psychological BulletinIntegration and generalization of kappas for multiple raters.
513 Citations1980Anthony J. Conger
BiometricsMeasuring Agreement between Two Judges on the Presence or Absence of a Trait
504 Citations1975Joseph L. Fleiss
Using the binary (positive-negative) case as a model, some of the proposed indexes for measuring agreement between two judges on a categorical scale are presented and critically evaluates.
Journal of Chronic DiseasesA comparison of Likert and visual analogue scales for measuring change in function
463 Citations1987Gordon Guyatt, Marie Townsend +2 more
The two methods of presenting response options show comparable responsiveness in a questionnaire measuring quality of life in chronic lung disease, and the ease of administration and interpretation of the seven-point scale recommend its use in clinical trials.
Archives of General PsychiatryReliability Studies of Psychiatric Diagnosis
445 Citations1981William M. Grove
The problems of research design and execution in studies of diagnostic reliability are reviewed, and statistical problems are examined, and solutions to many of these problems are suggested, including recommendations of appropriate reliability coefficients and data analyses.
JAMAThe Philosophical Basis of Peer Review and the Suppression of Innovation
416 Citations1990David F. Horrobin
The fundamental purpose of peer review in the biomedical sciences must be consistent with that of medicine itself, to cure sometimes, to relieve often, to comfort always.
New England Journal of MedicineThe Reliability of Clinical Methods, Data and Judgments
404 Citations1975Martín Keller
Statistical ScienceSelection Models and the File Drawer Problem
397 Citations1988Satish Iyengar, Joel B. Greenhouse
This paper uses selection models, or weighted distributions, to deal with one source of bias, namely the failure to report studies that do not yield statistically significant results, and applies selection models to two approaches that have been suggested for correcting the bias.
Psychological BulletinLarge sample variance of kappa in the case of different sets of raters.
386 Citations1979Joseph L. Fleiss, John Nee +1 more
PEDIATRICSObservation Scales to Identify Serious Illness in Febrile Children
362 Citations1982Paul L. McCarthy, Michael R. Sharpe +7 more
In the population studied, this predictive model, when used prior to history and physical examination, was reliable, predictive, specific, and sensitive for serious illness in febrile children.
Journal of Business ResearchMeta-analytic procedures for social research
349 Citations1994Elizabeth J. Wilson
Archives of General PsychiatryQuantification of Agreement in Psychiatric Diagnosis Revisited
349 Citations1987Patrick E. Shrout
It is argued that a new measure, Cohen's k statistic, 2 was the appropriate index of diagnostic agreement in psychiatry since it took into account the fact that raters agree by chance alone some of the time and only gave a perfect value if there was total agreement among the raters.
The British Journal of PsychiatryAssessing Inter-Rater Reliability for Rating Scales: Resolving some Basic Issues
322 Citations1976Domenic V. Cicchetti
The minimal sample sizes and the specific linear agreement weights required for assessing the reliability of rating scales commonly used in neuropsychiatric and other clinico-medical settings are presented.
The Journal of the Acoustical Society of AmericaIntensity Perception. II. Resolution in One-Interval Paradigms
295 Citations1972Louis D. Braida, N. I. Durlach
Biometrics2 x 2 Kappa Coefficients: Measures of Agreement or Association
293 Citations1989D. Blöch, Helena C. Kraemer
Two models, one for agreement and one for utility of association, are defined yielding different kappa coefficients and different sampling theory, and asymptotic results are derived for both models.
Applied Psychological MeasurementThe Effect of Number of Rating Scale Categories on Levels of Interrater Reliability : A Monte Carlo Investigation
291 Citations1985Domenic V. Cicchetti, Donald Shoinralter +1 more
A computer simulation study finds that reliability of a clinical scale increases steadily up to 7 scale points, beyond which no substantial increases occur, even when the number of scale poirits is increased to as many as 100.
Journal of Clinical EpidemiologyAgreement among reviewers of review articles
283 Citations1991Andrew D Oxman, Gordon Guyatt +5 more
Assessment of the consistency of an index of the scientific quality of research overviews found it was possible to achieve reasonable to excellent agreement for all of the items in the index, including the overall assessment of scientific quality.
Journal of Neurology Neurosurgery & PsychiatryThe brief scale for anxiety: a subdivision of the comprehensive psychopathological rating scale.
271 Citations1984Peter Tyrer, R T Owen +1 more
A rating scale suitable for recording anxious symptoms is described and comprises 10 items, all of which are rated on a 7 point scale, suitable for the rating of pathological anxiety alone or for anxiety occurring in the setting of other psychological or medical disorder.
American Journal of PsychiatryThe Structured Clinical Interview for DSM-III-R Dissociative Disorders: preliminary report on a new diagnostic instrument
243 Citations1990Marlene Steinberg, B Rounsaville +1 more
Preliminary findings from a study of 48 subjects with and without psychiatric diagnoses indicate good to excellent reliability and discriminant validity for the SCID-D as a diagnostic instrument for the five Dissociative disorders and as a tool for the evaluation of dissociative symptoms encountered within nondissociative syndromes.
The American Journal of MedicinePeer review in biomedical publication
242 Citations1974Franz J. Ingelfinger
Concurrence between two reviewers of each of some 500 papers submitted to the New England Journal of Medicine was only moderately better than a chance result, and the reviewing system was important to maintain standards.
PubMedA proposal for more informative abstracts of clinical articles. Ad Hoc Working Group for Critical Appraisal of the Medical Literature.
240 Citations1987Félix Gutzwiller, Fred Paccaud
Academy of Management JournalManuscript Characteristics Which Influence Acceptance for Management and Social Science Journals.
239 Citations1977S. J. Kerr, Joseph Thomas Tolliver +1 more
Contemporary Sociology A Journal of ReviewsJudgment Studies: Design, Analysis, and Meta-Analysis.
236 Citations1989Cecilia Ridgeway, Robert Rosenthal
American Sociological ReviewScholarly Consensus and Journal Rejection Rates
236 Citations1988Lowell L. Hargens
Annals of Internal MedicineA Proposal for More Informative Abstracts of Clinical Articles
235 Citations1987AD HOC WORKING GROUP FOR CRITICAL APPRAISAL OF THE MEDICAL LITERATURE
An easily implemented, although partial, solution is for authors of articles that have clinical implications to structure their abstracts so that key aspects of purpose, methods, and results are reported with a partly controlled vocabulary and in a standardized format.
BiometricsExtension of the Kappa Coefficient
235 Citations1980Helena C. Kraemer
An extension of the kappa coefficient is proposed which is appropriate for use with multiple observations per subject and for multiple response choices per observation and to illustrate new approaches to difficult problems in evaluation of reliability.
Sociological QuarterlyEditorial Policies and Practices Among Leading Journals in Four Scientific Fields
231 Citations1978Janice M. Beyer
PsychometrikaThe Relation between Information and Variance Analyses
222 Citations1956W. R. Garner, W. J. McGill
Journal of Experimental Social PsychologyBrilliant but cruel: Perceptions of negative evaluators
213 Citations1983Teresa M. Amabile
Applied Psychological MeasurementComparison of the Null Distributions of Weighted Kappa and the C Ordinal Statistic
206 Citations1977Dominic Cicchetti, Joseph L. Fleiss
Personality and Social Psychology BulletinCommitment and Voluntary Energy Conservation
198 Citations1976Michael S. Pallak, William H. Cummings
Psychosomatic MedicineAssessment of alexithymia in posttraumatic stress disorder and somatic illness: introduction of a reliable measure.
194 Citations1986John H. Krystal, Earl L. Giller +1 more
A novel measure of alexithymia, the Alexithymic Provoked Response Questionnaire (APRQ), showed a high degree of interrater reliability and a greater degree of correlation with the Beth Israel Psychosomatic Questionnaires (BIPQ) than a MMPI subscale or the Schalling‐Sifneos scale.
JAMAA Difficult Balance: Editorial Peer Review in Medicine
192 Citations1986Martha J. Carney
A comprehensive analysis of editorial peer review, looking at the merits and defects in the system from the viewpoint of many authorities is presented, making authors, editors, and reviewers think afresh about all the issues surrounding peer review in medical editing.
Journal of Applied Behavior AnalysisMEASURING THE RELIABILITY OF OBSERVATIONAL DATA: A REACTIVE PROCESS<sup>1</sup>
191 Citations1973Raymond G. Romanczyk, Ronald N. Kent +2 more
Reliability of observational data was measured simultaneously by two assessors under two experimental conditions, indicating that observers modified their observational criteria to approximate those of the identified assessor.
Applied Psychological MeasurementThe Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject
190 Citations1979Joseph L. Fleiss, Jack Cuzick
Applied Psychological MeasurementTesting the Normal Approximation and Minimal Sample Size Requirements of Weighted Kappa When the Number of Categories is Large
187 Citations1981Domenic V. Cicchetti
British Journal of Mathematical and Statistical PsychologyThe relativity of ‘absolute’ judgements
186 Citations1984Donald Laming
Journal of the American Medical AssociationTwins: A Study of Heredity and Environment
183 Citations1938
When you read more every page of this twins a study of heredity and environment, what you will obtain is something great.
The American StatisticianAssessment of 2 × 2 Associations: Generalization of Signal-Detection Methodology
181 Citations1988Helena C. Kraemer
Philosophy of Science and Its Discontents
180 Citations2019Steve Fuller
PubMedQuotational and reference accuracy in surgical journals. A continuing peer review problem.
167 Citations1990Jenny Evans, H I Nadjari +1 more
Journal of AdvertisingReplication in Advertising Research: 1977, 1978, 1979
165 Citations1981Leonard N. Reid, Lawrence C. Soley +1 more
American PsychologistImproving manuscript evaluation procedures.
160 Citations1972Donald D. Bowen, Robert Perloff +1 more
Convention program committees face a similar problem in selecting papers for presentation, although the published complaints usually focus exclusively on frustrations encountered in dealings with the journals.
Perception & PsychophysicsVariability and sequential effects in magnitude production and estimation of auditory intensity
148 Citations1977David M. Green, R. Duncan Luce +1 more
PsychometrikaSome New Results on Factor Indeterminacy
145 Citations1972Peter H. Schönemann, Mingmei Wang
PubMedA statistical analysis of reviewer agreement and bias in evaluating medical abstracts.
144 Citations1976Domenic V. Cicchetti, Harold O. Conn
Of the 77 abstracts rated for all parameters by all observers, interobserver agreement ranged between 81 and 88%.
Science Technology & Human ValuesConfirmational Response Bias Among Social Work Journals
143 Citations1990William Epstein
Psychological ReportsPsychology of Scientist: XXX. Credibility of Psychologists: An Empirical Study
142 Citations1970Leonard D. Goodstein, Karen Lee Brazis
JAMAGuarding the Guardians: A Conference on Editorial Peer Review
142 Citations1986Drummond Rennie
The system has improved and expanded to include many thousands of journals and an efficient apparatus for indexing and retrieving their contents, yet inquiries have so far failed to deal adequately with its most crucial part, peer review.
Journal of Applied Social PsychologyPublish or Politic: Referee Bias in Manuscript Review<sup>1</sup>
137 Citations1975Stephen I. Abramowitz, Beverly Gomes +1 more
The Journal of Experimental EducationThe Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals
119 Citations1989Herbert W. Marsh, Samuel A. Ball
Journal of Mathematical PsychologySome targets for mathematical psychology
114 Citations1975W. K. Estes
These problems might be alleviated by recognition of the complementary aspects of mathematical and computer simulation approaches to psychological theory and by a shift of emphasis from tactics to strategy in the construction and evaluation of models of both types.
Journal of Chronic DiseasesA proposed index for measuring agreement in test-retest studies
113 Citations1966Eugene Rogot, Irving D. Goldberg
Two indexes of agreement, A and A1, are compared by utilizing numerical examples and by application to published studies of reliability and the limitations of A1 are discussed.
Perception & PsychophysicsTwo tests of a neural attention hypothesis for auditory psychophysics
112 Citations1978R. Duncan Luce, David M. Green
A theory of neural attention is presented, and it is argued that intensity attention bands and critical bands of frequency are simply two manifestations of it that are studied by taking observations on the one variable while manipulating the other over a large range.
RadiologyHirschsprung disease: accuracy of the barium enema examination.
111 Citations1984Nancy S. Rosenfield, R C Ablow +5 more
It is concluded that the single most reliable radiographic sign of HD is the presence of a rectosigmoid transition zone, and the use of three radiographic features combined correlated better with the presence or absence of HD than did any of these features alone.
The Journal of the Acoustical Society of AmericaThe Information of Elementary Auditory Displays. II
107 Citations1953Irwin Pollack
Journal of Clinical InvestigationPeer review and publication. Presidential address before the 70th annual meeting of the American Society for Clinical Investigation, San Francisco, California, 30 April 1978.
105 Citations1978Jess Wilson
Psychological BulletinThe use and misuse of chi-square: Lewis and Burke revisited.
105 Citations1983Kevin Delucchi
Bulletin of the Psychonomic SocietyNew questions about old heritability estimates
103 Citations1989Peter H. Sch öNemann
ScientometricsAssessing precision in the manuscript review process: A little better than a dice roll
98 Citations1988Duncan Lindsey
The findings indicate that highly selective decision-making with imprecise reviewers results in outcomes that are only slightly better than chance.
Psychological MedicineComparative reliability of categorical and analogue rating scales in the assessment of psychiatric symptomatology
96 Citations1979Marina Remington, Peter Tyrer +2 more
The reliability of 26 items from the ninth edition of the Present State Examination was assessed using both the conventional categorical scales and separately constructed analogue scales, and the levels of inter-rater agreement obtained were comparable to those achieved in previous studies of PSE reliability.
…