Kappa Statistic

The kappa statistic is a measure of inter rater agreement for categorical data that adjusts for the amount of agreement expected to occur by chance alone. Cohen's kappa applies to two raters, while Fleiss' kappa extends the measure to three or more raters. Values range from negative 1 (systematic disagreement) through 0 (agreement no better than chance) to 1 (perfect agreement).

In research contexts where classification or categorization relies on human judgment, the degree to which independent assessors agree is a direct measure of the reliability of that judgment. Simple percent agreement overstates reliability because it ignores the fact that some agreement will occur by chance, particularly when the number of categories is small or when one category is much more common than others. Kappa addresses this limitation and provides a more honest assessment of agreement. It is commonly reported in systematic reviews (where two reviewers screen studies), diagnostic studies (where pathologists or radiologists independently classify cases), and behavioral research (where observers code actions).

Cohen's kappa for two raters is calculated by constructing a contingency table of rater classifications. Observed agreement (P_o) is the proportion of cases on which the two raters agree. Expected agreement (P_e) is the proportion of agreement that would be expected if raters assigned classifications independently, calculated from the marginal distributions of each rater's classifications. Kappa equals (P_o minus P_e) divided by (1 minus P_e).

For example, if two pathologists classify 200 tissue samples as malignant or benign and agree on 170 of them, P_o = 0.85. If the marginal distributions suggest that chance agreement would be 0.60, then kappa = (0.85 minus 0.60) divided by (1 minus 0.60) = 0.625, which counts as substantial agreement (0.61 to 0.80) on the widely used Landis and Koch (1977) benchmarks, a rule of thumb rather than a standard.

Weighted kappa is used when the categories are ordinal and some disagreements are "worse" than others. Linear weights assign increasing penalties for greater disagreements; quadratic weights penalize large disagreements proportionally more. The choice between linear and quadratic weights affects the value and should be reported alongside the result.

Kappa has known limitations. The kappa paradox refers to situations where high percent agreement produces low kappa values when the prevalence of one category is very high or very low. In these unbalanced situations, even small amounts of disagreement disproportionately reduce kappa because expected agreement is already high. Researchers should consider reporting both kappa and percent agreement, along with the prevalence of each category, to provide a complete picture.

Use Cohen's kappa whenever two raters independently classify items into nominal or ordinal categories and you need to assess agreement beyond chance. Use Fleiss' kappa when three or more raters are involved. Use weighted kappa when the categories are ordinal and disagreements have different severities. For continuous measurements, the intraclass correlation coefficient (ICC) is more appropriate. Kappa should be calculated during pilot phases of systematic reviews, diagnostic accuracy studies, and behavioral coding studies to calibrate raters and establish reliability before the main study begins.

A common error is interpreting kappa values using the standard benchmarks without considering the specific context. A kappa of 0.50 may be acceptable for a screening task with inherently ambiguous cases but unacceptable for a diagnostic classification with clear criteria. Another mistake is computing kappa on a very small sample of cases, which produces unstable estimates with wide confidence intervals. Researchers also sometimes fail to report the confidence interval for kappa, leaving readers unable to assess the precision of the reliability estimate. Ignoring the kappa paradox by reporting only kappa when prevalence is extreme can misrepresent the quality of agreement.

In a systematic review of interventions for chronic lower back pain, two reviewers independently screen 3,000 abstracts using predefined inclusion criteria. They agree on 2,820 abstracts (94% agreement), 450 included by both and 2,370 excluded by both; of the 180 disagreements, 100 were included by the first reviewer only and 80 by the second reviewer only. Cohen's kappa is 0.80 (95% CI: 0.77 to 0.83), at the top of the substantial band on the Landis and Koch benchmarks: chance agreement is already about 0.70, because both reviewers exclude most abstracts. The 180 discrepancies are resolved through discussion, and 12 require adjudication by a third reviewer.