Using Multilevel Random Coefficient Modeling to Investigate Rater Effects in Performance Ratings

Abstract

There has been recent interest in how rater attributes lead to systematic variance in ratings of job performance. Although numerous rater characteristics have been proposed to affect performance ratings, there has been little empirical research studying them. We suggest this has been because of methodological problems with levels of analysis and propose multilevel random coefficient (MRC) modeling as a solution. We present a multilevel model of rater effects in which ratees are nested within raters. We also present two examples of applying MRC modeling to criterion-related validity data to study how rater-level variables influence performance ratings and the relationships selection assessments have with those ratings.