Recent Issues Regarding rWG, rWG, rWG(J), and rWG(J)

Abstract

A recent article by Lindell and Brandt raised two concerns regarding the use of James, Demaree, and Wolf’s interrater agreement indices r WG and r WG(J) . First, they noted that the multi-item rWG(J) equation is mathematically equivalent to inserting r WG into the Spearman-Brown prophecy formula and questioned whether applying a formula developed for a reliability index was also appropriate for an agreement index. Second, they questioned the appropriateness of James et al.’s suggestion of replacing obtained negative values of r WG with zeros. This article addresses these concerns by demonstrating that r WG(J) can be derived independently from the Spearman-Brown prophecy formula and that negative values of r WG can be avoided by reparameterizing the structural equation underlying the data when there is systematic disagreement between subgroups of raters.