Interrater agreement for reviews for Developmental Review

Abstract

Reviewer reliability is examined for the first 2 years of operation of Developmental Review. Using the statistic, Finn's r, summary recommendations on publishability are shown to be higher in reliability than previously reported for other journals. Separate rating scales are shown to be less reliable than summary recommendations. Finn's r is compared with the intraclass correlation coefficient and shown to be a better method for examining reviewer agreement across different journals. Potential causes of high reviewer agreement for Developmental Review are discussed, and the importance of the distinction between reliability and validity of reviews is noted.