The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs

The results indicate that nonexperimental designs cannot be relied on to estimate the effectiveness of employment programs, and comparison group studies will be valid indicators of the program impacts.

Abstract

This study investigates empirically the strengths and limitations of using experimental versus nonexperimental designs for evaluating employment and training programs. The assessment involves comparing results from an experimental-design study-the National Supported Work Demonstration-with the estimated impacts of Supported Work based on analyses using comparison groups constructed from the Current Population Surveys. The results indicate that nonexperimental designs cannot be relied on to estimate the effectiveness of employment programs. Impact estimates tend to be sensitive both to the comparison group construction methodology and to the analytic model used. There is currently no way a priori to ensure that the results of comparison group studies will be valid indicators of the program impacts.