The debate on automated essay grading
IEEE Intelligent Systems and their ApplicationsPublished 1 September 2000
Marti A. Hearst
Citations172
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The director of the Natural Language Processing group at Educational Testing Service, Kukich, provides an insider's view of the history of the field of automated essay grading and describes how ETS is currently using computer programs to supplement human judges in the grading process.
Abstract
We look at a controversy: the use of computers for automated and semiautomated grading of exams. K. Kukich, the director of the Natural Language Processing group at Educational Testing Service, provides an insider's view of the history of the field of automated essay grading and describes how ETS is currently using computer programs to supplement human judges in the grading process. T. Landauer, D. Laham, and P. Foltz describe the use of latent semantic analysis in a commercial essay-scoring system called IEA. They also address important ethical questions. L. Hirschman, E. Breck, J. Burger, and L. Ferro report on MITRE's current efforts towards automated grading of short-answer questions and discuss the ramifications for the design of general question answering systems. Finally R. Calfee places these developments in the framework of current educational theory and practice.
Keywords
Computer Science
Journal of the American Society for Information ScienceIndexing by latent semantic analysis
12,677 Citations1990Scott Deerwester, Susan Dumais +3 more
Psychological ReviewA solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.
6,039 Citations1997Thomas K. Landauer, Susan Dumais
A new general theory of acquired similarity and knowledge representation, latent semantic analysis (LSA), is presented and used to successfully simulate such learning and several other psycholinguistic phenomena.
Discourse ProcessesAn introduction to latent semantic analysis
4,817 Citations1998Thomas K. Landauer, Peter W. Foltz +1 more
The adequacy of LSA's reflection of human knowledge has been established in a variety of ways, for example, its scores overlap those of humans on standard vocabulary and subject matter tests; it mimics human word sorting and category judgments; it simulates word‐word and passage‐word lexical priming data.
SIAM ReviewUsing Linear Algebra for Intelligent Information Retrieval
1,484 Citations1995Michael W. Berry, Susan Dumais +1 more
A lexical match between words in users’ requests and those in or assigned to documents in a database helps retrieve textual materials from scientific databases.
Psychological ReviewA solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.
853 Citations1997Thomas K. Landauer, Susan Dumais
Discourse ProcessesThe measurement of textual coherence with latent semantic analysis
750 Citations1998Peter W. Foltz, Walter Kintsch +1 more
The approach for predicting coherence through reanalyzing sets of texts from 2 studies that manipulated the coherence of texts and assessed readers’ comprehension indicates that the method is able to predict the effect of text coherence on comprehension and is more effective than simple term‐term overlap measures.
Deep Read
219 Citations1999Lynette Hirschman, Marc Light +2 more
Initial work on Deep Read, an automated reading comprehension system that accepts arbitrary text input (a story) and answers questions about it is described, with a baseline system that retrieves the sentence containing the answer 30--40% of the time.
Discourse ProcessesLearning from text: Matching readers and texts by latent semantic analysis
196 Citations1998Michael Wolfe, Michael Schreiner +5 more
Results show a nonmonotonic relation in which learning was greatest for texts that were neither too easy nor too difficult, and LSA proved as effective at predicting learning from these texts as traditional knowledge assessment measures.
Automated scoring using a hybrid feature identification technique
152 Citations1998Jill Burstein, Karen Kukich +5 more
This study exploits statistical redundancy inherent in natural language to automatically predict scores for essays using a hybrid feature identification method to score essay responses from test-takers of the Graduate Management Admissions Test (GMAT) and the Test of Written English (TWE).
Phi Delta KappanThe Computer Moves into Essay Grading: Updating the Ancient Test.
150 Citations1995Ellis B. Page, Nancy S. Petersen
A blind test has demonstrated that a computer can simulate the judgment of a group of human judges on a brand-new set of essays, and the bottom line is that the computer program did predict human judgments well - perhaps even better than three human judges.
Discourse ProcessesUsing latent semantic analysis to assess knowledge: Some technical considerations
149 Citations1998Bob Rehder, Michael Schreiner +4 more
This paper addresses four questions about how Latent Semantic Analysis achieves its results and how the results might be improved: what role does the use of technical vocabulary play?
Textbooks for Learning: Nurturing Children's Minds
148 Citations1998Marilyn J. Chambliss, Robert C. Calfee
This book presents a new approach to Textbook Design: Instructional Support Systems, which aims to bring about the Ideal by creating a Design for Publishing, Adopting and Selecting a Textbook.
Educational Measurement Issues and PracticeValidity and Automad Scoring: It's Not Only the Scoring
143 Citations1998Randy Elliot Bennett, Isaac I. Bejar
This research investigates the nature of the interplay among construct definition, task design, examinee interface, tutorial, test development tools, and automated scoring and reporting in the context of knowledge representation systems.
Digital Commons - University of South Florida (University of South Florida)Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil
127 Citations1997Michael Russell, Walt Haney
Findings show that, though multiple-choice test results do not differ much by mode of administration, for students accustomed to writing on computer, responses written on computer are substantially higher than those written by hand.
An unsupervised method for detecting grammatical errors
127 Citations2000Martin Chodorow, Claudia Leacock
An unsupervised method for detecting grammatical errors by inferring negative evidence from edited textual corpora and the errorrecognition system, ALEK, performs with about 80% precision and 20% recall.
IEEE Transactions on CommunicationsThe Writer's Workbench: Computer Aids for Text Analysis
125 Citations1982Nina H. Macdonald, Lawrence T. Frase +2 more
The Writer's Workbench programs, which analyze English prose and suggest improvements, are described, which include programs to proofread, comment on stylistic features of text, and provide reference information about the English language.
Language TestingThe prediction of TOEFL reading item difficulty: implications for construct validity
85 Citations1993Roy Freedle, Irene Kostin
Journal of Educational MeasurementA Tree‐Based Approach to Proficiency Scaling and Diagnostic Assessment
53 Citations1997Kathleen M. Sheehan
arXiv (Cornell University)How to Evaluate your Question Answering System Every Day and Still Get Real Work Done
40 Citations2000Eric Breck, John D. Burger +5 more
Text REtrieval ConferenceA Sys Called Qanda.
32 Citations1999Eric Breck, John D. Burger +4 more
The question answering system was built to experiment with natural language processing technologies such as shallow parsing, named entity tagging, and coreference chaining because it felt that the small number of terms in the questions coupled with the short length of the answers would make NLP technologies clearly beneficial.
Research in the Teaching of EnglishStudent Characteristics and Essay Test Writing Performance
11 Citations1983Sarah Warshauer Freedman
Toward Using Text Summarization for Essay-Based Feedback
4 Citations2000Jill Burstein, Ets Technologies +1 more
It is indicated that 40% and 60% discourse-based essay summaries improve the performance of the topical analysis module of e-rater, a system that electronically scores GMAT essays.