Evaluation of Parameter Estimation under Modified IRT Models and Small Samples.

This study both replicates and improves upon an earlier investigation into modified models into IRT Parameter Recovery, which found tentatively positive results.

Abstract

Accuracy of item parameter estimates is a critical concern for any application of item response theory (IRT). However, the necessary sample sizes are often difficult to obtain in practice, particularly for the more complex models. A promising avenue of research concerns modified item response models. This study both replicates and improves upon an earlier investigation into modified models (Parshall, Kromrey, and Chason, 1996), which found tentatively positive results. To obtain realistic data, empirical item parameters were generated by fitting a 6dimensional model to archival data, using NOHARM. These parameters were then used along with thetas generated from independent normal ability distributions to generate simulated item response data. One hundred datasets were generated for each of four sample sizes. Finally, BILOG was used to obtain estimated item and ability parameters for each of the six investigated models. Results were evaluated in terms of accuracy and stability across samples. Accuracy was assessed as the degree to which both the obtained item responses and the known response probabilities were reproduced from the generating parameters. Stability was assessed as empirical estimates of standard errors. Crossvalidation of fit and accuracy was accomplished by applying the sample item parameter estimates to additional samples generated from the same population. U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) Crrhis document has been reproduced as received from the person or organization originating it. Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position cr policy. to 1 CO CO N O PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY A-fAia_RoiC TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) 2 Paper presented at the annual meeting of the Psychometric Society, Gatlinburg, TN, June 26-29, I 1997. Modified IRT Models 2 Small Samples and Modified Models: An Investigation of IRT Parameter Recovery The advantages of item response theory (IRT) for testing have been discussed theoretically for a number of years (Hambleton & Swaminathan, 1985; Lord, 1980). The benefits to testing programs include applications for test development, equating, and computer adaptive testing. The most popular models applied in practice are the unidimensional 1parameter, 2-parameter, and 3-parameter logistic models (or, 1-PL, 2-PL, and 3-PL respectively). The formulas for these models are defined as