Evaluation of Clinical Competence: The Gap Between Expectation and Performance

The gaps between expectations and performance, and between data gathering and application, have important implications for institutional educational philosophy, suggesting a shift toward more clinically oriented and learner-directed strategies in the design of instructional and evaluation methods.

Abstract

OBJECTIVE: To evaluate a 3-year experience with the Objective Structured Clinical Examinations (OSCEs) and to compare faculty expectations with resident performance. DESIGN: Descriptive analysis of measures of resident performance. SETTING: Community-based pediatric residency program in Michigan. PARTICIPANTS: One hundred twenty-six pediatric residents at all levels of training. METHODS: The three examinations consisted of 36 to 42 5-minute stations, testing skills in physical examination, history, counseling, telephone management, and test interpretation. A committee of faculty and chief residents predetermined minimum pass levels for each resident level. Results were compared with other indices of resident performance. RESULTS: There was evidence for content, construct, and concurrent validity, as well as a high degree of reliability. However, 40% to 96% of residents scored below the minimum pass levels for their levels. In each examination, third-year residents had the highest failure rates, yet they scored well on the American Board of Pediatrics in-training examination and on their monthly clinical evaluations. Furthermore, for residents at all levels, the scores reflecting application of data were significantly lower than those assessing data gathering. CONCLUSIONS: The gaps between expectations and performance, and between data gathering and application, have important implications for institutional educational philosophy, suggesting a shift toward more clinically oriented and learner-directed strategies in the design of instructional and evaluation methods.