On Assessing Interrater Agreement for Multiple Attribute Responses
BiometricsPublished 1 September 1989
Lawrence L. Kupper, Kerry B. Hafner
Citations51
SJR quartileQ1
SJR score1.26
SNIP1.20
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A two-rater concordance statistic is derived, and associated statistical inference-making procedures are provided for assessing the extent of interrater agreement when each unit to be rated is characterized by a (possibly empty) subset of a specified set of distinct nominal attributes.
Abstract
New methods are developed for assessing the extent of interrater agreement when each unit to be rated is characterized by a (possibly empty) subset of a specified set of distinct nominal attributes. For such multiple attribute response data, a two-rater concordance statistic is derived, and associated statistical inference-making procedures are provided. This concordance statistic is corrected for chance agreement by using an underlying hypergeometric model. Numerical examples are given to illustrate the proposed methodology, and comparisons to other agreement statistics (e.g., kappa) are made.
Keywords
Decision SciencesMedicine
Wiley series in probability and statisticsStatistical Methods for Rates and Proportions
18,048 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
Society for Industrial and Applied Mathematics eBooksThe Jackknife, the Bootstrap and Other Resampling Plans
7,903 Citations1982Bradley Efron
Journal of the American Statistical AssociationEpidemiologic Research: Principles and Quantitative Methods.
2,942 Citations1984Lawrence S. Mayer, David G. Kleinbaum +2 more
The Journal of Nervous and Mental DiseaseON THE METHODS AND THEORY OF RELIABILITY
1,317 Citations1976John J. Bartko, William T. Carpenter
This paper reviews the most frequently used and misused reliability measures appearing in the mental health literature and suggests some suitable reliability measures to be used.
American Journal of EpidemiologyMISINTERPRETATION AND MISUSE OF THE KAPPA STATISTIC
816 Citations1987Malcolm Maclure, WC Willett
BiometricsMeasuring Agreement between Two Judges on the Presence or Absence of a Trait
504 Citations1975Joseph L. Fleiss
Using the binary (positive-negative) case as a model, some of the proposed indexes for measuring agreement between two judges on a categorical scale are presented and critically evaluates.
Archives of General PsychiatryQuantification of Agreement in Psychiatric Diagnosis Revisited
349 Citations1987Patrick E. Shrout
It is argued that a new measure, Cohen's k statistic, 2 was the appropriate index of diagnostic agreement in psychiatry since it took into account the fact that raters agree by chance alone some of the time and only gave a perfect value if there was total agreement among the raters.
BiometricsExtension of the Kappa Coefficient
235 Citations1980Helena C. Kraemer
An extension of the kappa coefficient is proposed which is appropriate for use with multiple observations per subject and for multiple response choices per observation and to illustrate new approaches to difficult problems in evaluation of reliability.
The British Journal of PsychiatryCoefficients of Agreement Between Observers and Their Interpretation
195 Citations1977Annette E. Maxwell
The measurement of agreement between ratings of patient symptomatology by two or more psychiatrists is discussed and coefficients which allow for possible chance agreement are to be preferred, but they involve assumptions about the way in which chance factors may operate.
British Journal of Mathematical and Statistical PsychologyMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
156 Citations1968B. S. Everitt
BiometricsThe Kappa Coefficient of Agreement for Multiple Observers When the Number of Subjects is Small
80 Citations1986Shulamith T. Gross
A modified kappa coefficient of agreement for multiple categories is proposed and a parameter-free distribution for testing null agreement is provided, for use when the number of raters is large relative to the numberof categories and subjects.
Archives of General PsychiatryQuantification of Agreement in Multiple Psychiatric Diagnosis
72 Citations1972Joseph L. Fleiss
To deal with the problem of indexing diagnostic agreement when clinicians make multiple diagnoses on the same case, an algorithm for calculating a specific level of disagreement for an individual case was developed, and a summary statistic for measuring overall agreement is defined.
International Journal of EpidemiologyBias and Efficiency in Logistic Analyses of Stratified Case-Control Studies
66 Citations1980Malcolm C. Pike, Alexandra Hill +1 more
There would seem to be strong arguments for the routine use of matched designs as the reduction in confounding bias produced by such matching may be considerable whereas there is only a small loss in efficiency compared with unmatched designs.
The Journal of the American Dental AssociationA clinical comparison of X-ray films for detection of proximal surface caries
36 Citations1985Mel L. Kantor, Allan B. Reiskin +1 more
It appears that Kodak Ektaspeed should be the preferred film for the dental practitioner because there is a 40% to 50% reduction in radiation exposure to the patient when the faster film is used.
BiometricsThe Large-Sample Relative Efficiency of the Mantel-Haenszel Estimator in the Fixed-Strata Case
30 Citations1986Allan Donner, Walter W. Hauck
The large-sample relative efficiency of the Mantel-Haenszel estimator psi MH of an odds ratio is investigated for the case in which the number of strata is fixed and the sample sizes within each stratum increase indefinitely.
BiometricsThe Asymptotic Relative Efficiency of the Mantel-Haenszel Estimator in the Increasing-Number-of-Strata Case
23 Citations1988Walter W. Hauck, Allan Donner
The Journal of Nervous and Mental DiseaseAgreement Measurement and the Judgment Process
20 Citations1979Cynthia L. Janes
This paper reviews the rationale for correcting for chance agreement between two raters and recommends Maxwell's Random Error Coefficient of Agreement (RE) as a measure to be used under conditions of equal proportions in the disagreement cells.
BiometricsEvaluating Interrater Agreement Through "Case-Control" Sampling
9 Citations1987Robert J. Jannarone, Caroline A. Macera +1 more
An approach to evaluating interrater agreement that is based on "case-control" sampling scheme, in which individuals are classified by one rating method and stratified prior to being classified by a second rating method, indicates that the new approach may be useful in practice.