Intercoder reliability indices: disuse, misuse, and abuse
Quality & QuantityPublished 28 November 2013
Guangchao Charles Feng
Citations108
SJR quartileQ1
SJR score0.87
SNIP1.85
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Although intercoder reliability has been considered crucial to the validity of a content study, the choice among them has been controversial. This study analyzed all the content studies published in the two major communication journals that reported intercoder reliability, aiming to find how scholars conduct intercoder reliability test. The results revealed that some intercoder reliability indices were misused persistently concerning the levels of measurement, the number of coders, and the means of reporting reliability over the past 30 years. Implications of misuse, disuse, and abuse were discussed, and suggestions regarding proper choice of indices in various situations were made at last. Copyright Springer Science+Business Media Dordrecht 2014
Keywords
Decision Sciences
PsychometrikaCoefficient Alpha and the Internal Structure of Tests
43,211 Citations1951Lee J. Cronbach
Journal of the American Statistical AssociationContent Analysis: An Introduction to its Methodology.
24,566 Citations1984Mack Shelley, Klaus Krippendorff
Wiley series in probability and statisticsStatistical Methods for Rates and Proportions
18,048 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
BiometricsA Concordance Correlation Coefficient to Evaluate Reproducibility
8,705 Citations1989Lawrence I-Kuei Lin
A new reproducibility index is developed and studied that is simple to use and possesses desirable properties and the statistical properties of this estimate can be satisfactorily evaluated using an inverse hyperbolic tangent transformation.
Psychological BulletinWeighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
8,537 Citations1968Jacob Cohen
The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi.
Journal of Clinical EpidemiologyHigh agreement but low Kappa: I. the problems of two paradoxes
2,886 Citations1990Alvan R. Feinstein, Domenic V. Cicchetti
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance.
Human Communication ResearchContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
2,886 Citations2002Matthew Lombard, Jennifer Snyder‐Duch +1 more
A content analysis of 200 studies utilizing content analysis published in the communication literature between 1994 and 1998 is used to characterize practices in the field and demonstrate that mass communication researchers often fail to assess (or at least report) intercoder reliability and often rely on percent agreement, an overly liberal index.
Analyzing media messages using quantitative content analysis in research
2,263 Citations2019Daniel Riffe, Stephen Lacy +1 more
Journal of Clinical EpidemiologyBias, prevalence and kappa
2,106 Citations1993Ted Byrt, Janet Bishop +1 more
New indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices.
British Journal of Mathematical and Statistical PsychologyComputing inter‐rater reliability and its variance in the presence of high agreement
1,932 Citations2006Kilem L. Gwet
This paper explores the origin of these limitations, and introduces an alternative and more stable agreement coefficient referred to as the AC1 coefficient, and proposes new variance estimators for the multiple-rater generalized pi and AC1 statistics, whose validity does not depend upon the hypothesis of independence between raters.
Journal of Clinical EpidemiologyHigh agreement but low kappa: II. Resolving the paradoxes
1,836 Citations1990Domenic V. Cicchetti, Alvan R. Feinstein
An omnibus index offers a single summary expression for a fourfold table of binary concordance among two observers and the paradoxes of kappa are desirable since they appropriately "penalize" inequalities in ppos and pneg.
Public Opinion QuarterlyReliability of Content Analysis: The Case of Nominal Scale Coding
1,836 Citations1955William Scott
Journal of Applied Communication ResearchRethinking validity and reliability in content analysis
1,610 Citations1999W. James Potter, Deborah Levine‐Donnerstein
It is argued that validity and reliability should be conceptualized differently across the various forms of content and the various uses of theory, and that content analyses need not be limited to theory‐based coding schemes and standards set by experts.
Computational LinguisticsInter-Coder Agreement for Computational Linguistics
1,604 Citations2008Ron Artstein, Massimo Poesio
It is argued that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.
Educational and Psychological MeasurementCoefficient Kappa: Some Uses, Misuses, and Alternatives
1,544 Citations1981Robert L. Brennan, Dale J. Prediger
Journal of Consumer ResearchContent-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity
1,526 Citations1991Richard H. Kolbe, Melissa S. Burnett
Journal of Marketing ResearchReliability of Nominal Data Based on Qualitative Judgments
1,326 Citations1989William D. Perreault, Laurence E. Leigh
ScholarlyCommons (University of Pennsylvania)Computing Krippendorff's Alpha-Reliability
815 Citations2011Klaus Krippendorff
Krippendorff’s alpha () is a reliability coefficient developed to measure the agreement among observers, coders, judges, raters, annotators or measuring instruments drawing distinctions among typically unstructured phenomena or assign computable values to them.
Psychological BulletinIntegration and generalization of kappas for multiple raters.
513 Citations1980Anthony J. Conger
Wiley series in probability and statisticsThe Measurement of Interrater Agreement
512 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
Communication Methods and MeasuresAgreement and Information in the Reliability of Coding
429 Citations2011Klaus Krippendorff
The concept of reliability, simple agreement, three conceptions of chance to correct that agreement, and sources of information deficiency are explored, and two measures of information about reliability, akin to the power of a statistical test, are developed.
Public Opinion QuarterlyCommunications Through Limited Response Questioning
247 Citations1954Eugene Bennett, Robyn Alpert +1 more
Sociological MethodologyBivariate Agreement Coefficients for Reliability of Data
235 Citations1970Klaus Krippendorff
Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity
199 Citations2002Kilem L. Gwet
The British Journal of PsychiatryCoefficients of Agreement Between Observers and Their Interpretation
195 Citations1977Annette E. Maxwell
The measurement of agreement between ratings of patient symptomatology by two or more psychiatrists is discussed and coefficients which allow for possible chance agreement are to be preferred, but they involve assumptions about the way in which chance factors may operate.
Journal of Marketing ResearchIntercoder Reliability Estimation Approaches in Marketing: A Generalizability Theory Framework for Quantitative Data
164 Citations1990Marie Adele Hughes, Dennis E. Garrett
A review of estimation approaches for intercoder reliability reported in articles in leading marketing journals reveals that most marketing researchers are using inadequate measures, and recommends that marketing researchers report dependability indices based on generalizability theory for quantitative coding systems.
Journal of Biopharmaceutical StatisticsA Unified Approach for Assessing Agreement for Continuous and Categorical Data
113 Citations2007Lawrence Lin, A. Hedayat +1 more
Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions ofvariance components.
Communication Yearbook 36
65 Citations2012Charles T. Salmon
Journal of ClassificationA Formal Proof of a Paradox Associated with Cohen’s Kappa
63 Citations2010Matthijs J. Warrens
A formal proof of Cohen’s kappa penalizes judges with similar marginals compared to judges who produce different marginals is presented.
Quality & QuantityFactors affecting intercoder reliability: a Monte Carlo experiment
43 Citations2012Guangchao Charles Feng
Quality & QuantityUnderlying determinants driving agreement among coders
22 Citations2012Guangchao Charles Feng
With a Monte Carlo simulation, the determinants of the agreement indices were empirically tested and can be classified into a class based on the number of categories, the marginal distribution and the level of difficulty.
HKBU Institutional Repository (Hong Kong Baptist University)A Reliability index(ai) that assumes honest coders and variable randomness
5 Citations2012Xinshu Zhao