Using Cluster Analysis to Facilitate Standard Setting

Abstract

Abstract Setting standards on tests remains an important and pervasive problem in educational and psychological testing. Traditional standard-setting methods have been criticized due to reliance on untested subjective judgment, lack of demonstrated reliability, and lack of external validation. In this article, we present a new procedure designed to help improve previous standard-setting methods. This procedure involves cluster analyzing test takers to discover examinee groups useful for (a) envisioning marginally competent performance as required in test-centered standard-setting methods or (b) defining borderline or contrasting groups used in examinee-centered methods. We applied the procedure to a state-wide mathematics proficiency test. The standards derived from the cluster analyses were compared with those established at the local level and with those derived from a more traditional borderline and contrasting groups analysis. We observed relative congruence across the local cutscores and those derived using cluster analysis, and we observed similar correlations among the resulting proficiency groupings and course grades. The results of the more traditional borderline and contrasting groups analyses were less favorable. We conclude that cluster analysis appears useful for helping set standards on educational tests. Suggestions for future research are provided.