Measurement of Observer Agreement
RadiologyPublished 1 August 2003
Harold L. Kundel, Marcia Polansky
Citations1,486
SJR quartileQ1
SJR score3.86
SNIP3.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The review concentrates on the chance-corrected indices, kappa and weighted kappa, which are used in diagnostic imaging for expressing observer agreement in regard to categorical data.
Abstract
Statistical measures are described that are used in diagnostic imaging for expressing observer agreement in regard to categorical data. The measures are used to characterize the reliability of imaging methods and the reproducibility of disease classifications and, occasionally with great care, as the surrogate for accuracy. The review concentrates on the chance-corrected indices, kappa and weighted kappa. Examples from the imaging literature illustrate the method of calculation and the effects of both disease prevalence and the number of rating categories. Other measures of agreement that are used less frequently, including multiple-rater kappa, are referenced and described briefly.
Keywords
Decision SciencesMedicine
BiometricsThe Measurement of Observer Agreement for Categorical Data
78,934 Citations1977J. Richard Landis, Gary G. Koch
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Wiley series in probability and statisticsStatistical Methods for Rates and Proportions
18,048 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
BiometricsAn Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers
3,808 Citations1977J. Richard Landis, Gary G. Koch
A subset of 'observers who demonstrate a high level of interobserver agreement can be identified by using pairwise agreement statistics betweeni each observer and the internal majority standard opinion on each subject.
Journal of Clinical EpidemiologyHigh agreement but low Kappa: I. the problems of two paradoxes
2,886 Citations1990Alvan R. Feinstein, Domenic V. Cicchetti
In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance.
Journal of Clinical EpidemiologyHigh agreement but low kappa: II. Resolving the paradoxes
1,836 Citations1990Domenic V. Cicchetti, Alvan R. Feinstein
An omnibus index offers a single summary expression for a fourfold table of binary concordance among two observers and the paradoxes of kappa are desirable since they appropriately "penalize" inequalities in ppos and pneg.
New England Journal of MedicineVariability in Radiologists' Interpretations of Mammograms
598 Citations1994Joann G. Elmore, Carolyn K. Wells +3 more
Efforts to improve accuracy and reduce variability in interpretation may increase the effectiveness of mammography in detecting early breast cancers.
American Journal of RoentgenologyBreast imaging reporting and data system standardized mammography lexicon: observer variability in lesion description.
181 Citations1996Jay A. Baker, Phyllis J. Kornguth +1 more
BI-RADS is moderately successful in providing a standardized language for physicians to describe lesion morphology and efforts to reevaluate specific terms and the diagnostic significance assigned to calcification descriptors may prove useful in maintaining the promise of improved quality with the BI-R ADS standardized mammography lexicon.
British Journal of RadiologyVariation between experienced observers in the interpretation of accident and emergency radiographs.
162 Citations1999P Robinson, D.R. Wilson +3 more
The magnitude of interobserver variation in plain film reporting is considerable, and must be taken into account when designing assessment techniques and setting quality standards for this activity.
Journal of the American Medical AssociationTUBERCULOSIS CASE FINDING
144 Citations1947C. C. Birkelo
Within a relatively short time photofluorography has been transformed from a crude experiment to an efficient diagnostic aid and the outlook for the future is that mass radiography will further expand and that a majority of the adult population will receive this service.
Medical Decision MakingReceiver Operator characteristic (ROC) Analysis without Truth
109 Citations1990R. Mark Henkelman, Ian Kay +1 more
The method is shown to give results equivalent to conventional ROC analysis in a comparison of computed tomography, radionuclide scintigraphy, and magnetic resonance imaging for liver metastasis.
American Journal of RoentgenologyAccuracy of Screening Mammography Using Single Versus Independent Double Interpretation
62 Citations2000Stephen H. Taplin, Carolyn M. Rutter +4 more
An analysis among 31 community radiologists to identify the average change in screening mammography interpretive accuracy afforded by independent double interpretation foundindependent double interpretation does not increase accuracy as measured by the area under the ROC curve.
Investigative RadiologyModeling Approaches for the Analysis of Observer Agreement
56 Citations1992John S. Uebersax
RadiologyInterobserver variation in the interpretation of abdominal radiographs.
52 Citations1989J B Markus, Sat Somers +3 more
A total of 140 sets of abdominal radiographs were reviewed independently by four qualified diagnostic radiologists and there was fair to excellent interobserver agreement for 11 signs and diagnoses and poor agreement for the remaining eight.
Investigative RadiologyThe Effect of Verification on the Assessment of Imaging Techniques
48 Citations1990George Révész, Harold L. Kundel +1 more
New England Journal of MedicineThe Accuracy of Mammographic Interpretation
46 Citations1994Daniel B. Kopans
It should come as no surprise that there is a range of skills and expertise among physicians involved in similar activities, and this is true for radiologists who inte...
Skeletal RadiologyObserver variation in the detection of osteopenia
41 Citations1986David Epstein, Murray K. Dalinka +4 more
Investigative RadiologyThe Effect of Verification on the Assessment of Imaging Techniques
41 Citations1983George Révész, Harold L. Kundel +1 more
To assess relative accuracies of three different radiographic techniques, chest radiographs were taken of 66 patients with each of the three techniques, and the films were evaluated by six radiologists, finding that any one of the techniques could be shown to be better than the other, depending on how the truth was defined.
RadiologyLimited correlation of left ventricular end-diastolic pressure with radiographic assessment of pulmonary hemodynamics.
38 Citations1990Peter G. Herman, Aftab Khan +4 more
Left ventricular end-diastolic pressure (LVEDP) is a reliable indicator of the diastolic function of the left ventricle and correlation between radiographic signs of CHF and LVEDP was limited.
RadiologyAccuracy of bedside chest hard-copy screen-film versus hard- and soft-copy computed radiographs in a medical intensive care unit: receiver operating characteristic analysis.
27 Citations1997H L Kundel, Warren B. Gefter +5 more
The results provide some justification for using bedside chest computed radiography and for reading soft-copy images from a high-quality display.
Investigative RadiologyImproving Diagnostic Accuracy: A Comparison of Interactive and Delphi Consultations
26 Citations1977Bruce J. Hillman, Samuel J. Hessel +2 more
To assess the relative effectiveness of independent decision making, interactive group consultation, and Delphi techniques in a clinical setting, the diagnostic accuracy of 17 radiologists interpreting radiologic examinations in these settings was compared.
Academic RadiologyMixture distribution and receiver operating characteristic analysis of bedside chest imaging with screen-film and computed radiography
16 Citations1997Harold L. Kundel, Marcia Polansky
Mixture distribution analysis may be useful for comparing imaging techniques in situations where the true imaging diagnosis cannot be established with a method independent of that being evaluated.
Acta RadiologicaEstimation or quantification of tumour volume? CT study on irregular phantoms
12 Citations2001M. Tiitola, L. Kivisaari +5 more
Subjective size estimation of irregular tumours should be repeatedly performed by the same observer, or by using the mean or median estimate of several observers, especially for film readers with limited radiological experience.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Comparing observer performance with mixture distribution analysis when there is no external gold standard</title>
7 Citations1998Harold L. Kundel, Marcia Polansky
The study utilizing a data set consisting of independent, blinded readings by 4 radiologists of a stratified sample of 95 bedside chest images obtained using computed radiography concludes that the MDA may be a viable alternative to the ROC for evaluating observer performance on imaging modalities in clinical settings where image verification is either difficult or impossible.
Journal of the American Medical AssociationTHE "PERSONAL EQUATION" IN THE INTERPRETATION OF A CHEST ROENTGENOGRAM
5 Citations1947
SPIE eBooksAgreement and Accuracy Mixture Distribution Analysis
5 Citations2010Marcia Polansky
It is essential to assess the accuracy of diagnostic tests, that is, the likelihood that a reader is correct with respect to an established standard, to ensure that the diagnoses of the study cases are accurate.
Acta RadiologicaESTIMATION OR QUANTIFICATION OF TUMOUR VOLUME?. CT study on irregular phantoms
2 Citations2001M. Tiitola, L. Kivisaari +5 more