Modelling patterns of agreement and disagreement

A survey of ways of statistically modelling patterns of observer agreement and disagreement is presented, with main emphasis on modelling inter-observer agreement for categorical responses, both for nominal and ordinal response scales.

Abstract

This article presents a survey of ways of statistically modelling patterns of observer agreement and disagreement. Main emphasis is placed on modelling inter-observer agreement for categorical responses, both for nominal and ordinal response scales. Models discussed include (1) simple cell-probability models based on Cohen's kappa that focus on beyond-chance agreement, (2) loglinear models for square tables, such as quasi-independence and quasi-symmetry models, (3) latent class models that express the joint distribution between ratings as a mixture of clusters for homogeneous subjects, each cluster having the same 'true' rating, and 4) Rasch models, which decompose subject-by-observer rating distributions using observer and subject main effects. Models can address two distinct components of agreement--strength of association between ratings, and similarity of marginal distributions of the ratings.