Using computer-aided assessment to test higher level learning outcomes

This paper sets out an approach using a revised Bloom's taxonomy of learning objectives for the careful design of objective questions to assist in the assessment of higher learning outcomes (HLO’s) and details the creation and evaluation of a variety of such questions.

Abstract

This paper sets out an approach using a revised Bloom's taxonomy of learning\nobjectives for the careful design of objective questions to assist in the assessment of\nhigher learning outcomes (HLO’s) and details the creation and evaluation of a\nvariety of such questions. This has been done within the context of two constraints;\nthe use of popular, commercially available computer-aided assessment (CAA)\nsoftware, specifically Question Mark Perception and Half-Baked Hot Potatoes, and\nthe assumption of limited learning technologist support. It examines the problems\ninherent in the design of objective questions for HLO’s (specifically at the levels of\nApplication, Analysis and Evaluation) and introduces a framework by which\nsystematic design may be carried out. It examines the mode of design, construction\nand evaluation of 22 such objective questions devised for formative assessment for\ntwo post-graduate course units in Information Systems, involving two groups of\nstudents, 37 in total. The results from the trialing of these questions raises issues of\ncrucial differences in CAA software for feedback, scoring and delivery. The paper\nexamines key statistical indicators of question quality (facility and discrimination\nindices), as well as the results of interviews with students, to drawn conclusions\nabout the best use of question types, student preparation for tests, and discusses\nkey issues in question preparation. It concludes with an examination of the main\nadvantages and disadvantages of using CAA for HLO’s referring to the resource\noverheads needed and the problems inherent in the process.