Measuring Agreement between Two Judges on the Presence or Absence of a Trait

Using the binary (positive-negative) case as a model, some of the proposed indexes for measuring agreement between two judges on a categorical scale are presented and critically evaluates.

Abstract

At least a dozen indexes have been proposed for measuring agreement between two judges on a categorical scale. Using the binary (positive-negative) case as a model, this paper presents and critically evaluates some of these proposed measures. The importance of correcting for chance-expected agreement is emphasized, and identities with intraclass correlation coefficients are pointed out.