Inter Rater Reliability

Inter rater reliability is the degree of agreement between two or more independent assessors who evaluate, rate, or classify the same set of items, participants, or data. It is commonly measured using Cohen's kappa statistic for two raters, Fleiss' kappa for three or more raters, or the intraclass correlation coefficient (ICC) for continuous ratings.

Many research activities require subjective judgment: screening studies for inclusion in a systematic review, rating the quality of evidence, coding qualitative data, diagnosing conditions from medical images, or scoring behavioral observations. If different assessors reach different conclusions when evaluating the same material, the results of the study depend on who happens to do the assessment rather than on the underlying data. High inter rater reliability provides assurance that the measurement process is consistent and that findings are not artifacts of individual assessor bias. Cochrane's conduct standards require at least two people to decide independently whether each study meets the eligibility criteria, and many review teams report kappa or percent agreement for screening and risk of bias assessment, although Cochrane does not require a kappa statistic to be calculated.

The most widely used measure for categorical ratings by two raters is Cohen's kappa, which adjusts observed agreement for the amount of agreement expected by chance alone. The formula is kappa = (observed agreement minus expected agreement) divided by (1 minus expected agreement). Kappa values range from negative 1 (systematic disagreement) through 0 (agreement no better than chance) to 1 (perfect agreement). The conventional benchmarks of Landis and Koch (1977) interpret values below 0 as poor, 0 to 0.20 as slight, 0.21 to 0.40 as fair, 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial, and 0.81 to 1.00 as almost perfect, though these labels are rules of thumb rather than rigid standards.

When more than two raters are involved, Fleiss' kappa extends the calculation to accommodate multiple assessors. For ordinal or continuous ratings, weighted kappa assigns partial credit for disagreements that are close to agreement (for example, one rater assigns a score of 3 and another assigns 4), and the ICC provides a continuous measure of consistency or absolute agreement depending on the model chosen.

Several factors affect inter rater reliability. The clarity and specificity of the coding scheme or assessment tool are paramount: vague definitions lead to inconsistent application. Rater training improves reliability by aligning assessors' understanding of the criteria. Pilot testing the assessment on a calibration sample before the main study allows discrepancies to be identified and resolved. The prevalence of the condition being assessed also affects kappa; when the condition is very common or very rare, kappa can be low even when agreement is high (the "kappa paradox"), because expected agreement by chance is also high.

Measure inter rater reliability whenever a study involves subjective judgment by multiple assessors. This includes systematic review screening (where two reviewers independently assess each record), quality assessment (where raters apply tools such as the Cochrane risk of bias tool), diagnostic studies (where radiologists or pathologists independently interpret images or specimens), behavioral research (where observers code behavior from video recordings), and qualitative research (where coders apply thematic codes to interview transcripts). Report the reliability measure, the number of assessors, the number of items assessed, and the resolution process for disagreements.

A common error is reporting percent agreement without adjusting for chance, which inflates the apparent level of agreement. Simple percent agreement ignores the fact that some agreement will occur by luck, especially when categories are few. Another mistake is failing to train raters adequately before the main assessment, leading to low reliability that compromises the study. Some researchers calculate kappa on a very small calibration sample and then proceed with single rater assessment for the remainder, which does not establish that reliability was maintained throughout the study. Finally, using inappropriate reliability measures for the data type (for example, using Cohen's kappa for continuous ratings instead of the ICC) can produce misleading results.

Two independent reviewers screen 2,400 titles and abstracts for a systematic review on interventions for adolescent depression. After independent screening, they agree on 2,180 records (91% agreement). Cohen's kappa is calculated as 0.78, indicating substantial agreement. The 220 records on which they disagree are discussed, and 15 require resolution by a third reviewer. The team reports the kappa value and the resolution process in the methods section of the published review.