Distributional Ratings of Performance: More Evidence for a New Rating Format

Abstract

We examined the extent to which performance ratings obtained using a distributional rating format differ from ratings obtained using a more typical global rating scale format. We sought to determine whether a distributional format actually elicits different information than a global format and the extent to which measures of within performance variability derived from distributional ratings are related to levels of interrater agreement on global rating scales. Confirmatory factor analyses of three models of measurement equivalence indicated that a model of tau-equivalence in which the two rating formats provided similar measurements but with differing levels of measurement error provided the best fit to the data. Further, we found a large, significant relationship between measures of within dimension performance variabili5 and interrater agreement. Our results suggest that a distributional rating format is a viable alternative rating format that may provide a richer source of information than do more typical global rating scales.