A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target

Abstract

The commonly used form of r wg. (J) can display irregular behavior, so four variants of this index were examined. An alternative index, r* wg. J , is recommended. This index is an inverse linear function of the ratio of the average obtained variance to the variance of uniformly distributed random error. r* wg.J is superficially similar to Cronbach’s α, but careful examination confirms that r* wg.J is an index of agreement, not reliability. Based on an examination of the small-sample behavior of r wg and r* wg.J , sample sizes of 10 or more raters are recommended.