Rethinking critical issues in performance assessment

The authors call for a critical appraisal of applying Classical Test Theory to the assessment of job-related behaviors and discuss several critical issues: the meanings of reliability and validity in performance assessment and the balance between them, stability of performance, and the role of experts in clinical performance assessment.

Abstract

The recent interest of medical schools and licensure organizations in establishing performance assessment methods in medical education presents new challenges to medical educators. The problems encountered in establishing the reliability and validity properties of performance assessment necessitate rethinking the accepted definitions of reliability and validity. The authors examine the relationship between Classical Test Theory and job performance. They discuss several critical issues: the meanings of reliability and validity in performance assessment and the balance between them (including simple versus complex behaviors), stability of performance, specific versus generic abilities, and the role of experts in clinical performance assessment. The authors call for a critical appraisal of applying Classical Test Theory to the assessment of job-related behaviors.