Raters who pursue different goals give different ratings.

Significant multiple correlations are found, both within classes and in an analysis of the pooled sample, between rating goals obtained after raters had observed a significant proportion of ratee performance and teacher evaluations collected at the end of the semester.

Abstract

J. N. Cleveland and K. R. Murphy (1992) suggested that phenomena such as rater errors and interrater disagreements could be understood in terms of differences in the goals pursued by various raters. We measured 19 rating goals of students at the beginning of a semester, grouped them into scales, and correlated these with teacher evaluations collected at the end of the semester. We found significant multiple correlations, both within classes and in an analysis of the pooled sample (adjusting for instructor mean differences, incremental R2 =.08). Measures of rating goals obtained after raters had observed a significant proportion of ratee performance accounted for variance (incremental R2 =.07) not accounted for by measures of goals obtained at the beginning of the semester.