Standard Setting: The Next Generation (Where Few Psychometricians Have Gone Before!)

Abstract

Abstract Two major testing practices that emerged in the decade of the 1990s-the use of polytomous item formats and multiple cut-scores-have stimulated a new generation of standard-setting methods. These methods are reviewed in the context of national and state testing programs. The best strategies from the decade of the 1980s with a proven track record are then combined with the most promising new techniques into a Generic Eclectic Method (GEM) for standard setting. This GEM provides a structure and 10-step iterative, behavioral-anchoring judgmental process that can be applied to almost any educational, licensure, or certification test. Decisions on six issues are required to use GEM in a specific cut-score situation: (a) examinee target population (students, teachers, administrators, etc.), (b) unit of judgment (item, item cluster, or work sample), (c) item scoring format (dichotomous, polytomous, or a combination of both), (d) test-centered (unscored unit) or examinee-centered (previously scored unit or item response theory scale) approach, (e) number of achievement levels (cut-scores), and (f) optional weighting of objectives for decision policy analysis. Various types of reliability and validity evidence of the effectiveness of the judgmental process are described, and directions for future research are suggested.