Beyond kappa: A review of interrater agreement measures
Canadian Journal of StatisticsPublished 1 March 1999
Mousumi Banerjee, Michelle C. Capozzoli, Laura A. McSweeney, Debajyoti Sinha
Citations979
SJR quartileQ2
SJR score0.59
SNIP0.91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Abstract In 1960, Cohen introduced the kappa coefficient to measure chance‐corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence‐interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them.
Keywords
Decision Sciences
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
BiometrikaLongitudinal data analysis using generalized linear models
18,123 Citations1986Kung‐Yee Liang, Scott L. Zeger
Wiley series in probability and statisticsStatistical Methods for Rates and Proportions
18,048 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
BiometricsA Concordance Correlation Coefficient to Evaluate Reproducibility
8,705 Citations1989Lawrence I-Kuei Lin
A new reproducibility index is developed and studied that is simple to use and possesses desirable properties and the statistical properties of this estimate can be satisfactorily evaluated using an inverse hyperbolic tangent transformation.
Psychological BulletinWeighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
8,537 Citations1968Jacob Cohen
The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi.
Journal of Clinical EpidemiologyHigh agreement but low Kappa: I. the problems of two paradoxes
2,886 Citations1990Alvan R. Feinstein, Domenic V. Cicchetti
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance.
Springer series in statisticsMeasures of Association for Cross Classifications
2,260 Citations1979Leo A. Goodman, William Kruskal
Journal of Clinical EpidemiologyBias, prevalence and kappa
2,106 Citations1993Ted Byrt, Janet Bishop +1 more
New indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices.
Public Opinion QuarterlyReliability of Content Analysis: The Case of Nominal Scale Coding
1,836 Citations1955William Scott
Psychological BulletinLarge sample standard errors of kappa and weighted kappa.
1,504 Citations1969Joseph L. Fleiss, Jacob Cohen +1 more
Journal of the American Statistical AssociationSimple Models for the Analysis of Association in Cross-Classifications Having Ordered Categories
831 Citations1979Leo A. Goodman
American Journal of EpidemiologyMISINTERPRETATION AND MISUSE OF THE KAPPA STATISTIC
816 Citations1987Malcolm Maclure, WC Willett
Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical CharacterI. Mathematical contributions to the theory of evolution. —VII. On the correlation of characters not quantitatively measurable
473 Citations1900Karl Pearson
BiometricsA One-Way Components of Variance Model for Categorical Data
468 Citations1977J. Richard Landis, Gary G. Koch
Statistical Methods in Medical ResearchReview papers : Design and analysis of reliability studies
326 Citations1992Graham Dunn
This review covers the design and analysis of essentially two types of reliability study: method comparison studies and generalizability experiments, which are expected to be used to supplement the simpler traditional methods.
Biometrics2 x 2 Kappa Coefficients: Measures of Agreement or Association
293 Citations1989D. Blöch, Helena C. Kraemer
Two models, one for agreement and one for utility of association, are defined yielding different kappa coefficients and different sampling theory, and asymptotic results are derived for both models.
Statistics in MedicineA goodness‐of‐fit approach to inference procedures for the kappa statistic: Confidence interval construction, significance‐testing and sample size estimation
258 Citations1992Allan Donner, Michael Eliasziw
A new procedure for constructing a confidence interval about the kappa statistic in the case of two raters and a dichotomous outcome is proposed, based on a chi-square goodness-of-fit test as applied to a model frequently used for clustered binary data.
BiometricsExtension of the Kappa Coefficient
235 Citations1980Helena C. Kraemer
An extension of the kappa coefficient is proposed which is appropriate for use with multiple observations per subject and for multiple response choices per observation and to illustrate new approaches to difficult problems in evaluation of reliability.
Statistics in MedicineMeasuring interrater reliability among multiple raters: An example of methods for nominal data
228 Citations1990Karen L. Posner, Paul D. Sampson +3 more
Modifications of previously published estimators appropriate for measurement of reliability in the case of stratified sampling frames are introduced and interpret these measures in view of standard errors computed using the jackknife.
Applied Psychological MeasurementComparison of the Null Distributions of Weighted Kappa and the C Ordinal Statistic
206 Citations1977Dominic Cicchetti, Joseph L. Fleiss
British Journal of Mathematical and Statistical PsychologyMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
156 Citations1968B. S. Everitt
Journal of the American Statistical AssociationModeling Agreement among Raters
156 Citations1985Martin A. Tanner, Michael A. Young
By examining a hierarchy of log-linear models, it is shown how one can analyze the agreement among the raters in a manner analogous to the analysis of association in a contingency table.
Statistical Methods in Medical ResearchModelling patterns of agreement and disagreement
146 Citations1992Alan Agresti
A survey of ways of statistically modelling patterns of observer agreement and disagreement is presented, with main emphasis on modelling inter-observer agreement for categorical responses, both for nominal and ordinal response scales.
Applied Psychological MeasurementInference About Weighted Kappa in the Non-Null Case
142 Citations1978Joseph L. Fleiss, Domenic V. Cicchetti
Statistics in MedicineLatent class analysis of diagnostic agreement
131 Citations1990John S. Uebersax, William M. Grove
This work describes refinements in the estimation of parameters for varying panel designs, and applies latent class methods successfully to examples of medical agreement data that include data previously found to be poorly fitted by two-class models.
BiometricsThe Maximum Likelihood Estimation of Correlation from Contingency Tables
127 Citations1962G. M. Tallis
BiometricsMaximum Likelihood Estimation of Agreement in the Constant Predictive Probability Model, and Its Relation to Cohen's Kappa
122 Citations1990Mikel Aickin
The parameters of the corresponding constant predictive probability model are shown to be estimable by the method of maximum likelihood, and a simulation study indicates applicability of the asymptotic results to finite samples.
Annals of Otology Rhinology & LaryngologyDistribution and Heritability of Recurrent Ear Infections
115 Citations1997Kari Jorunn Kværner, Jennifer R. Harris +2 more
The distribution of recurrent ear infections was obtained from a population-based sample of 2,750 pairs of Norwegian twins and the relative contribution of genes and environment to variability in the predisposition to develop otitis media was estimated by means of structural equation modeling.
Journal of Clinical EpidemiologyThe statistical analysis of kappa statistics in multiple samples
93 Citations1996Allan Donner, Neil Klar
Partitioning methods allow a variety of hypotheses to be tested, including an assessment of the degree of agreement within each sample, a testing procedure based on the pooled data, and a test of heterogeneity that may be used to assess the validity of pooling across samples.
BiometricsA Weighted Concordance Correlation Coefficient for Repeated Measurement Designs
86 Citations1996Vernon M. Chinchilli, Juliann K. Martel +2 more
This work develops weighted product-moment and concordance correlation coefficients which are applicable for repeated measurements study designs and considers two distinct situations in which the repeated measurements are paired or unpaired over time.
BiometricsGeneral Observer-Agreement Measures on Individual Subjects and Groups of Subjects
80 Citations1984Dianne O’Connell, Annette J. Dobson
BiometricsTesting the Homogeneity of Kappa Statistics
74 Citations1996Allan Donner, Michael Eliasziw +1 more
Two procedures are compared for testing the homogeneity of k > or = 2 independent kappa statistics in the case of two raters and a dichotomous outcome based on the estimated large sample variance derived under a model frequently adopted for inferences concerning interobserver agreement.
Sociological MethodologyA Stabilized Newton-Raphson Algorithm for Log-Linear Models for Frequency Tables Derived by Indirect Observation
73 Citations1988Shelby J. Haberman
A new algorithm is proposed that converges both more quickly and more reliably than currently available alternatives in estimation of asymptotic variances of parameter estimates and may be applied to both grouped and ungrouped data.
Statistics in MedicineA comparison of methods for calculating a stratified kappa
61 Citations1991William E. Barlow, Mei‐Ying Lai +1 more
Acta Psychiatrica ScandinavicaFamilial influences on the clinical characteristics of major depression: a twin study
60 Citations1992K. S. Kendler, Michael C. Neale +3 more
The results suggest that familial factors influence the predisposition to some clinical features of MD, and twin resemblance for the degree of impairment and neurovegetative symptoms are probably due largely to genetic factors.
Research in Nursing & HealthKappa muddles together two sources of disagreement: Tetrachoric correlation is preferable
59 Citations1993T. P. Hutchinson
When assessing agreement between experts, it is important to distinguish between disagreements that can and cannot be explained by different placing of the boundaries between categories.
PubMedQuasi-symmetric latent class models, with application to rater agreement.
56 Citations1993Alan Agresti, Joseph B. Lang
American Journal of EpidemiologyJACKKNIFING FUNCTIONS OF MULTINOMIAL FREQUENCIES, WITH AN APPLICATION TO A MEASURE OF CONCORDANCE1
55 Citations1982Joseph L. Fleiss, Mark Davies
Journal of EconometricsLatent variable models for ordered categorical data
53 Citations1983David J. Bartholomew
PubMedLatent variable models for clustered ordinal data.
51 Citations1995Yinsheng Qu, Marion R. Piedmonte +1 more
BiometricsQuasi-Symmetric Latent Class Models, with Application to Rater Agreement
46 Citations1993Alan Agresti, Boby Joseph
This work considers latent class models for the joint classification that satisfy quasi-symmetry when subject-specific response distributions are such that for a given subject, responses on different variables are independent, and odds ratios comparing marginal distributions of the variables are identical for each subject.
PubMedThe epidemiology of pregnancy complications and outcome in a Norwegian twin population.
45 Citations1992Linda A. Corey, K Berg +2 more
Maternal genetic factors make an important contribution to a predisposition for dizygotic twinning, contribute to the risk of miscarriage, and appear to determine, in part, whether a woman experiences nauseavomiting or hypertension-toxemia during pregnancy.
BiometricsLatent Variable Models for Clustered Dichotomous Data with Multiple Subclusters
44 Citations1992Yinsheng Qu, George W. Williams +2 more
BiometricsMeasurement of Interrater Agreement with Adjustment for Covariates
40 Citations1996William E. Barlow
The kappa coefficient measures chance-corrected agreement between two observers in the dichotomous classification of subjects and assumes both raters have the same marginal probability of classification, but this probability may depend on one or more covariates.
Statistics in MedicineHow many raters? toward the most reliable diagnostic consensus
35 Citations1992Helena C. Kraemer
Comparisons and contrasts between the known results with an interval consensus and those with a binary consensus are drawn and tactics for use in a pilot study are suggested to answer the question can a consensus diagnosis be used to improve reliability?
BiometrikaThe equivalence of tetrachoric and maximum likelihood estimates of p in 2 × 2 tables
33 Citations1970M. A. Hamdan
Statistics in MedicineEstimating kappa from binocular data
31 Citations1991Neal L. Oden
A method to estimate kappa, a measure of agreement between two graders, when both graders rate the same set of pairs of eyes, is discussed, which makes use of the correlated binocular data to estimate confidence intervals for the common kappa.
Archives of General PsychiatryWhat Is the 'Right' Statistical Measure of Twin Concordance (or Diagnostic Reliability and Validity)?
31 Citations1997H C Kraemer
Statistics in MedicineModelling covariate effects in observer agreement studies: The case of nominal scale agreement
29 Citations1995Patrick Graham
The proposed methodology facilitates investigation of issues such as confounding of covariate effects and interaction between covariates in their effect on chance-corrected agreement and allows observer agreement data to be analyzed in a manner strongly analogous to the logistic modelling of the association between disease and suspected risk factors.
Statistics in MedicineEstimating kappa from binocular data and comparing marginal probabilities
26 Citations1993Hubert J. A. Schouten
An alternative procedure is proposed that utilizes existing formulae for the computation of (weighted) kappa and its standard error and the McNemar test is generalized to compare the probabilities of a positive judgement by the two graders.
PubMedAssessing interrater agreement from dependent data.
25 Citations1997John Williamson, Amita K. Manatunga
Statistics in MedicineMaximum likelihood estimation of the kappa coefficient from models of matched binary responses
23 Citations1995Mohamed M. Shoukri, Stephen Martin +1 more
An estimate of the kappa-coefficient of agreement between two methods of rating based on matched pairs of binary responses is presented and it is shown that the estimate depends on the common intraclass correlation coefficient between the pairs.
Statistics in MedicineA HIERARCHICAL APPROACH TO INFERENCES CONCERNING INTEROBSERVER AGREEMENT FOR MULTINOMIAL DATA
15 Citations1997Allan Donner, Michael Eliasziw
A new method based on a series of nested, statistically independent inferences, each corresponding to a binary outcome variable obtained by combining a substantively relevant subset of the original categories is proposed.
Statistics in MedicineExposure—response relationship for a dichotomized response when the continuous underlying variable is not measured
7 Citations1988L. Irwig, H. T. Groeneveld
This work presents a method of using the biserial correlation coefficient and normal distribution theory to estimate exposure-response relationships at any required threshold for each observer, and correcting for random observational error using the reliability coefficient.