Judging Relative Importance: Direct Rating and Point Allocation Are Not Equivalent

This series of experiments investigates two commonly used methods of assigning numerical values to attributes in order to signify their perceived relative importance, from which Rating and PA, in different ways, exhibit consistent elicitation-dependent bias.

Abstract

In this series of experiments we investigate two commonly used methods of assigning numerical values (i.e., decision weights) to attributes in order to signify their perceived relative importance. The two methods are to ask people to directly rate each of the attributes in turn (Rating), or to allocate a budget of points (typically 100 points) to the attributes (Point Allocation or PA). These procedures may seem to be minor variants of one another, yet they produce very different profiles of decision weights. The differences are predicted by a simple, idealized model of weighting, from which Rating and PA, in different ways, exhibit consistent elicitation-dependent bias.