CAA of Short Non-MCQ Answers

The expected results from prototyping using ATM are obtained, indicating the reliability and feasibility of this new approach for the detailed assessment of text contents incorporating word order.

Abstract

This paper presents a new approach for the computer-assisted assessment (CAA)\nof non- multiple choice questions (Non-MCQ) type and short answers given by\nstudents. The technique is developed for the assessment of text contents of free text\nanswers to questions of factual disciplines.\nThe Automated Text Marker (ATM) prototype automatically breaks down an expertly\nwritten model answer, to a closed-ended question, into the smallest viable unit of\nconcepts with their dependencies accounted for by automatically tagging the\nresultant concepts and their dependencies with numbers. The same process is\napplied to each student’s answer and the resultant concepts and their\ndependencies are then pattern-matched with those of the model examiner’s answer.\nTwo main components of ATM are the syntax and semantics analysers. In a\nprototype test, ATM provides for one score for the grammars and the other for the\ntext contents.\nThe focus of this paper is on semantic analysis of text contents since the syntactic\nanalysis of sentences has been generally and successfully automated.\nVarious examples of sentences of different factual disciplines such as those of\nProlog programming, psychology and biology-related fields are analysed.\nJustifications for these analyses of sentences are provided and the corresponding\nprototype tests are conducted. The expected results from prototyping using ATM are\nobtained, indicating the reliability and feasibility of this new approach for the detailed\nassessment of text contents incorporating word order. Work is currently underway for building a larger and more comprehensive ATM\nsystem for analysing and assessing text components larger than sentences such as\nparagraphs and whole text passages. Unlike existing computerised assessment\nsystems, ATM is not a predictive system, although, like a human assessor, it is not\nperfect.