Factors affecting intercoder reliability: a Monte Carlo experiment
Quality & QuantityPublished 12 September 2012
Guangchao Charles Feng
Citations43
SJR quartileQ1
SJR score0.87
SNIP1.85
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Although it has long been a consensus that intercoder reliability is crucial to the validity of a content analysis study, the choice among them has been debated. This study reviewed and empirically tested most popular intercoder reliability indices, aiming to find the most robust index against prevalence and rater bias, by empirically testing their relationships with response surface methodology through a Monte Carlo experiment. It was found that Maxwell’s R.E is superior to Krippendorff’s α, Scott’s π, Cohen’s κ, I r of Perreault and Leigh, and Gwet’s AC 1 . More nuanced relationships among prevalence, sensitivity, specificity and the intercoder reliability indices were discovered through response surface plots. Both theoretical and practical implications were also discussed in the end. Copyright Springer Science+Business Media B.V. 2013
Keywords
Decision SciencesMathematics
Revue économiqueNonparametric Statistics for the Behavioral Sciences
35,492 Citations1958Henri Guitton, Sidney Siegel
Journal of the American Statistical AssociationContent Analysis: An Introduction to its Methodology.
24,566 Citations1984Mack Shelley, Klaus Krippendorff
Psychological BulletinIntraclass correlations: Uses in assessing rater reliability.
23,020 Citations1979Patrick E. Shrout, Joseph L. Fleiss
TechnometricsResponse Surface Methodology: Process and Product Optimization Using Designed Experiments
9,581 Citations1996Richard F. Gunst, Raymond H. Myers +1 more
Using a practical approach, this book discusses two-level factorial and fractional factorial designs, several aspects of empirical modeling with regression techniques, focusing on response surface methodology, mixture experiments and robust design techniques.
BiometricsA Concordance Correlation Coefficient to Evaluate Reproducibility
8,705 Citations1989Lawrence I-Kuei Lin
A new reproducibility index is developed and studied that is simple to use and possesses desirable properties and the statistical properties of this estimate can be satisfactorily evaluated using an inverse hyperbolic tangent transformation.
Psychological BulletinWeighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
8,537 Citations1968Jacob Cohen
The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi.
Contemporary Sociology A Journal of ReviewsDiscrete Multivariate Analysis: Theory and Practice.
5,160 Citations1975James R. Beniger, Yvonne Bishop +4 more
PsychometrikaNote on the Sampling Error of the Difference Between Correlated Proportions or Percentages
4,283 Citations1947Quinn McNemar
Two formulas are presented for judging the significance of the difference between correlated proportions and the chi square equivalent of one of the developed formulas.
Physical TherapyThe Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements
4,081 Citations2005Julius Sim, Chris Wright
The issue of statistical testing of kappa is considered, including the use of confidence intervals, and appropriate sample sizes for reliability studies using kappa are tabulated.
Communication Methods and MeasuresAnswering the Call for a Standard Reliability Measure for Coding Data
4,002 Citations2007Andrew F. Hayes, Klaus Krippendorff
This work proposes Krippendorff's alpha as the standard reliability measure, general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data.
Journal of Clinical EpidemiologyHigh agreement but low Kappa: I. the problems of two paradoxes
2,886 Citations1990Alvan R. Feinstein, Domenic V. Cicchetti
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance.
The Journal of Nervous and Mental DiseaseNONPARAMETRIC STATISTICS FOR THE BEHAVIORAL SCIENCES
2,234 Citations1957Siegel Sidney
Journal of Clinical EpidemiologyBias, prevalence and kappa
2,106 Citations1993Ted Byrt, Janet Bishop +1 more
New indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices.
British Journal of Mathematical and Statistical PsychologyComputing inter‐rater reliability and its variance in the presence of high agreement
1,932 Citations2006Kilem L. Gwet
This paper explores the origin of these limitations, and introduces an alternative and more stable agreement coefficient referred to as the AC1 coefficient, and proposes new variance estimators for the multiple-rater generalized pi and AC1 statistics, whose validity does not depend upon the hypothesis of independence between raters.
Journal of Clinical EpidemiologyHigh agreement but low kappa: II. Resolving the paradoxes
1,836 Citations1990Domenic V. Cicchetti, Alvan R. Feinstein
An omnibus index offers a single summary expression for a fourfold table of binary concordance among two observers and the paradoxes of kappa are desirable since they appropriately "penalize" inequalities in ppos and pneg.
Public Opinion QuarterlyReliability of Content Analysis: The Case of Nominal Scale Coding
1,836 Citations1955William Scott
Computational LinguisticsInter-Coder Agreement for Computational Linguistics
1,604 Citations2008Ron Artstein, Massimo Poesio
It is argued that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.
Educational and Psychological MeasurementCoefficient Kappa: Some Uses, Misuses, and Alternatives
1,544 Citations1981Robert L. Brennan, Dale J. Prediger
Journal of Marketing ResearchReliability of Nominal Data Based on Qualitative Judgments
1,326 Citations1989William D. Perreault, Laurence E. Leigh
Wiley series in probability and statisticsStatistical Design and Analysis of Experiments
983 Citations2003Robert Mason, Richard F. Gunst +1 more
Canadian Journal of StatisticsBeyond kappa: A review of interrater agreement measures
979 Citations1999Mousumi Banerjee, Michelle C. Capozzoli +2 more
Choice Reviews OnlineStatistical design and analysis of experiments: with applications to engineering and science
957 Citations1990
People have look hundreds of times for their chosen readings like this statistical design and analysis of experiments with applications to engineering and science, but end up in infectious downloads.
ScholarlyCommons (University of Pennsylvania)Computing Krippendorff's Alpha-Reliability
815 Citations2011Klaus Krippendorff
Krippendorff’s alpha () is a reliability coefficient developed to measure the agreement among observers, coders, judges, raters, annotators or measuring instruments drawing distinctions among typically unstructured phenomena or assign computable values to them.
Critical CareStatistics review 13: receiver operating characteristic curves.
790 Citations2004Viv Bewick, Liz Cheek +1 more
This review introduces some commonly used methods for assessing the performance of a diagnostic test and the uses of the receiver operating characteristic curve.
Latex : A Document Preparation System
716 Citations1985Leslie Lamport
This user's guide and reference for the LaTeX computer typesetting system has been revised to document features available in release LaTeX2e.
Wiley series in probability and statisticsResponse Surfaces, Mixtures, and Ridge Analyses
603 Citations2006George E. P. Box, Norman R. Draper
BiometricsA Note On "A Concordance Correlation Coefficient to Evaluate Reproducibility"
519 Citations1997Carol A. Nickerson
Psychological BulletinIntegration and generalization of kappas for multiple raters.
513 Citations1980Anthony J. Conger
Wiley series in probability and statisticsThe Measurement of Interrater Agreement
512 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
BiometricsMeasuring Agreement between Two Judges on the Presence or Absence of a Trait
504 Citations1975Joseph L. Fleiss
Using the binary (positive-negative) case as a model, some of the proposed indexes for measuring agreement between two judges on a categorical scale are presented and critically evaluates.
Journal of Clinical EpidemiologyA rEAPPRAISAL OF THE KAPPA COEFFICIENT
500 Citations1988William Thompson, Stephen D. Walter
It is concluded that under certain circumstances kappa can be interpreted as an indicator of validity, i.e. unbiasedness of the odds ratio, rather than simply as one of reliability.
Archives of General PsychiatryA Proposed Solution to the Base Rate Problem in the Kappa Statistic
488 Citations1985Edward L. Spitznagel
A statistic for the quantification of agreement is proposed (the Y statistic), which can be related to kappa but which is completely independent of prevalence in the case of validity studies and relatively so in the cases of reliability.
BiometrikaA TEST FOR HOMOGENEITY OF THE MARGINAL DISTRIBUTIONS IN A TWO-WAY CLASSIFICATION
462 Citations1955Alan Stuart
COMPSTATSweave: Dynamic Generation of Statistical Reports Using Literate Data Analysis
434 Citations2002Friedrich Leisch
Sweave combines typesetting with LATEX and data anlysis with S into integrated statistical documents that can be automatically updated if data or analysis change, which allows truly reproducible research.
Elsevier eBooksInterrater Reliability and Agreement
378 Citations2000Howard E. A. Tinsley, David J. Weiss
Archives of General PsychiatryQuantification of Agreement in Psychiatric Diagnosis Revisited
349 Citations1987Patrick E. Shrout
It is argued that a new measure, Cohen's k statistic, 2 was the appropriate index of diagnostic agreement in psychiatry since it took into account the fact that raters agree by chance alone some of the time and only gave a perfect value if there was total agreement among the raters.
Journal of Clinical EpidemiologyBehavior and interpretation of the κ statistic: Resolution of the two paradoxes
312 Citations1996Charles A. Lantz, Elliott Nebenzahl
Psychological MethodsRater bias in psychological research: When is it a problem and what can we do about it?
302 Citations2000William T. Hoyt
Using a model based on multivariate generalizability theory, a model identifies 4 types of bias that may affect findings in research using observer ratings, including the biases traditionally termed leniency and halo errors.
PsychometrikaRamifications of a Population Model for <i>κ</i> as a Coefficient of Reliability
295 Citations1979Helena C. Kraemer
Biometrics2 x 2 Kappa Coefficients: Measures of Agreement or Association
293 Citations1989D. Blöch, Helena C. Kraemer
Two models, one for agreement and one for utility of association, are defined yielding different kappa coefficients and different sampling theory, and asymptotic results are derived for both models.
Journal of Clinical EpidemiologyBias and prevalence effects on kappa viewed in terms of sensitivity and specificity
278 Citations2000Fred K. Hoehler
It is suggested that investigators concentrate on obtaining populations with trait prevalence near 50% rather than searching for statistical indices to rescue or excuse inefficient experiments.
Public Opinion QuarterlyCommunications Through Limited Response Questioning
247 Citations1954Eugene Bennett, Robyn Alpert +1 more
Psychological BulletinDiversity of decision-making models and the measurement of interrater agreement.
247 Citations1987John S. Uebersax
The British Journal of PsychiatryComparing the Classification of Subjects by Two Independent Judges
243 Citations1970A. E. Maxwell
Sociological MethodologyBivariate Agreement Coefficients for Reliability of Data
235 Citations1970Klaus Krippendorff
Journal of Biomedical InformaticsMeasuring agreement in medical informatics reliability studies
229 Citations2002George Hripcsak, Daniel F. Heitjan
It is helpful to separate the components of disagreement when the goal is to improve the reliability of an instrument or of the raters, and kappa may underestimate the true agreement and observed agreement may overestimate it.
Journal of the American Statistical AssociationA Note on the Equivalence of Two Test Criteria for Hypotheses in Categorical Data
225 Citations1966V. P. Bhapkar
Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity
199 Citations2002Kilem L. Gwet
The British Journal of PsychiatryCoefficients of Agreement Between Observers and Their Interpretation
195 Citations1977Annette E. Maxwell
The measurement of agreement between ratings of patient symptomatology by two or more psychiatrists is discussed and coefficients which allow for possible chance agreement are to be preferred, but they involve assumptions about the way in which chance factors may operate.
Journal of Clinical EpidemiologyThe dependence of Cohen's kappa on the prevalence does not matter
178 Citations2005Werner Vach
Journal of the American Statistical AssociationModeling Agreement among Raters
156 Citations1985Martin A. Tanner, Michael A. Young
By examining a hierarchy of log-linear models, it is shown how one can analyze the agreement among the raters in a manner analogous to the analysis of association in a contingency table.
Statistical Methods in Medical ResearchModelling patterns of agreement and disagreement
146 Citations1992Alan Agresti
A survey of ways of statistically modelling patterns of observer agreement and disagreement is presented, with main emphasis on modelling inter-observer agreement for categorical responses, both for nominal and ordinal response scales.
Statistical Tools for Measuring Agreement
138 Citations2011Lawrence Lin, A. Hedayat +1 more
This paper presents a basic approach for paired continuous data when target values are random or fixed and a unified approach for continuous and categorical data.
Statistics in MedicineLatent class analysis of diagnostic agreement
131 Citations1990John S. Uebersax, William M. Grove
This work describes refinements in the estimation of parameters for varying panel designs, and applies latent class methods successfully to examples of medical agreement data that include data previously found to be poorly fitted by two-class models.
Journal of Clinical EpidemiologyThe meaning of kappa: Probabilistic concepts of reliability and validity revisited
122 Citations1996I. Guggenmoos-Holzmann
A framework--the "agreement concept"--is developed to study the use of Cohen's kappa as well as alternative measures of chance-corrected agreement in a unified manner and it is demonstrated that for 2 x 2 tables an adequate choice can be made only if the characteristics of the observational setting are taken into account.
Understanding StatisticsInterrater Agreement Measures: Comments on Kappa<sub>n</sub>, Cohen's Kappa, Scott's π, and Aickin's α
120 Citations2003Louis M. Hsu, Ronald H. Field
Journal of Biopharmaceutical StatisticsA Unified Approach for Assessing Agreement for Continuous and Categorical Data
113 Citations2007Lawrence Lin, A. Hedayat +1 more
Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions ofvariance components.
The British Journal of PsychiatryCoefficients of Agreement
99 Citations1983Michael Dewey
This paper criticizes the proposed random error coefficient (RE) and suggests that Cohen's (1960) coefficient kappa (K) is still the measure of choice.
Statistics in MedicineHOW reliable are change‐corrected measures of agreement?
85 Citations1993I. Guggenmoos-Holzmann
It is concluded that improvements in the design and the analysis of reliability studies are a prerequisite for valid and pertinent results.
Journal of Evaluation in Clinical PracticeRelationships between statistical measures of agreement: sensitivity, specificity and kappa
74 Citations2008Martin Feuerman, Allen R. Miller
The analytic formulas and graph could be potentially useful to clinicians and biostatisticians in better interpreting the outcomes of an alternative diagnostic test whenever the measures sensitivity, specificity and kappa are employed together.
Human Communication ResearchA Call for Standardization in Content Analysis Reliability
69 Citations2004Matthew Lombard
Multivariate Behavioral ResearchOn Generalizations Of The G Index And The Phi Coefficient To Nominal Scales
67 Citations1979Svante Janson, Jan Vegelius
If the same categories are used for two nominal scale variables, this information should be used in similarity measures between those variables, and two such similarity measures, one proposed by Goodman & Kruskal and one (kappa) by Cohen are examined.
Journal of ClassificationA Formal Proof of a Paradox Associated with Cohen’s Kappa
63 Citations2010Matthijs J. Warrens
A formal proof of Cohen’s kappa penalizes judges with similar marginals compared to judges who produce different marginals is presented.
Statistics in MedicineA comparison of methods for calculating a stratified kappa
61 Citations1991William E. Barlow, Mei‐Ying Lai +1 more
Research in Nursing & HealthKappa muddles together two sources of disagreement: Tetrachoric correlation is preferable
59 Citations1993T. P. Hutchinson
When assessing agreement between experts, it is important to distinguish between disagreements that can and cannot be explained by different placing of the boundaries between categories.
PubMedQuasi-symmetric latent class models, with application to rater agreement.
56 Citations1993Alan Agresti, Joseph B. Lang
BiometricsQuasi-Symmetric Latent Class Models, with Application to Rater Agreement
46 Citations1993Alan Agresti, Boby Joseph
This work considers latent class models for the joint classification that satisfy quasi-symmetry when subject-specific response distributions are such that for a given subject, responses on different variables are independent, and odds ratios comparing marginal distributions of the variables are identical for each subject.
European PsychologistOn the Marginal Dependency of Cohen’s κ
43 Citations2008Alexander von Eye, Maxine von Eye
Journal of Clinical EpidemiologyModelling observer agreement—an alternative to kappa
40 Citations1994Sherryl M. May
Methodology for the modelling of interobserver agreement has been developed and the details and advantages are demonstrated by way of the analysis of three examples.
Bayesian Methods for Measures of Agreement
36 Citations2009Lyle D. Broemeling
The Bayesian Approach to Model-Based Approaches Agreement and Matching Agreement and Correlated Observations and Modeling and Agreement Agreement with Quantitative Scores are presented.
Biometrical JournalRaking Kappa: Describing Potential Impact of Marginal Distributions on Measures of Agreement
36 Citations1995Alan Agresti, Atalanta Ghosh +1 more
This paper introduces a smoothed version of kappa computed after raking the table to achieve pre-specified marginal distributions, and compares it with raked kappa for various margins to indicate the extent of the dependence on the margins.
Journal of Consulting and Clinical PsychologyMeasuring diagnostic agreement.
35 Citations1996James Langenbucher, Erich W. Labouvie +1 more
British Journal of Sports MedicineBasic statistics and epidemiology: a practical guide
31 Citations2005A Ugoni
The author’s writing style is engaging and very easy to follow, which prevents the book from becoming a required text for either students or practitioners of epidemiological research, as the author's writing style is engaging and very easy to follow.
EpidemiologySensitivity and Specificity-like Measures of the Validity of a Diagnostic Test That Are Corrected for Chance Agreement
23 Citations1992Steven S. Coughlin, Linda W. Pickle
The British Journal of PsychiatryAn Extension of the Random Error Coefficient of Agreement to N x N Tables
16 Citations1979Cynthia L. Janes
A more general formula for RE's computation is presented and it is shown how the new form is applicable to cases with more than two response categories.
Biometrical JournalModels of Chance when Measuring Interrater Agreement with Kappa
15 Citations1991Alexander von Eye, Silvia Sörensen
Statistics in Biopharmaceutical ResearchBayesian and Frequentist Methodologies for Analyzing Method Comparison Studies With Multiple Methods
13 Citations2010Pankaj K. Choudhary, Kunshan Yin
HKBU Institutional Repository (Hong Kong Baptist University)When to use Scott’s π or Krippendorff's α, if ever?
2 Citations2011Xinshu Zhao
It is shown that π or α should be used only when (a) coders enforce a predetermined quota as the first priority and (b)Coders maximize chance coding as the second priority, which means that neither can be a general indicator.
…