AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0
ETS Research Report SeriesPublished 1 December 2004Open access
Yigal Attali, Jill Burstein
Citations523
SJR quartileQ3
SJR score0.36
SNIP0.79
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new version of e-rater is described that differs from the previous one with regard to the feature set and model building approach and evidence on the validity and reliability of scores produced by the new version is presented.
Abstract
ABSTRACT The e‐rater® system has been used by ETS for automated essay scoring since 1999. This paper describes a new version of e‐rater (v.2.0) that differs from the previous one (v.1.3) with regard to the feature set and model building approach. The paper describes the new version, compares the new and previous versions in terms of performance, and presents evidence on the validity and reliability of scores produced by the new version.
Keywords
Computer Science
Communications of the ACMA vector space model for automatic indexing
7,434 Citations1975Gerard Salton, Anita M.-Y. Wong +1 more
An approach based on space density computations is used to choose an optimum indexing vocabulary for a collection of documents, demonstating the usefulness of the model.
Discourse ProcessesAn introduction to latent semantic analysis
4,817 Citations1998Thomas K. Landauer, Peter W. Foltz +1 more
The adequacy of LSA's reflection of human knowledge has been established in a variety of ways, for example, its scores overlap those of humans on standard vocabulary and subject matter tests; it mimics human word sorting and category judgments; it simulates word‐word and passage‐word lexical priming data.
IEEE Transactions on Pattern Analysis and Machine IntelligenceUsing discriminant eigenfeatures for image retrieval
1,630 Citations1996D.L. Swets, Juyang Weng
This paper describes the automatic selection of features from an image training set using the theories of multidimensional discriminant analysis and the associated optimal linear projection, and demonstrates the effectiveness of these most discriminating features for view-based class retrieval from a large database of widely varying real-world objects.
Medical Entomology and ZoologySignal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers
784 Citations1996John A. Swets
The Relative Operating Characteristic in Psychology is studied as a form of Empirical ROCs in Discrimination and Diagnostic Tasks and Applications in Various Diagnostic Fields.
The Journal of Experimental EducationComputer Grading of Student Prose, Using Modern Concepts and Software
227 Citations1994Ellis B. Page
This research analyzed recent federal samples of 495 and 599 essays and simulated groups of human judges, reaching multiple Rs as high as .87, close to the apparent reliability of the targeted judge groups.
IEEE Intelligent SystemsFinding the WRITE stuff: automatic identification of discourse structure in student essays
191 Citations2003Jill Burstein, Daniel Marcu +1 more
The discourse analysis software, which is embedded in Criterion, an online essay evaluation application, uses machine learning to identify discourse elements in student essays that exemplify how teachers perform this task.
Choice Reviews OnlineConstruction versus choice in cognitive measurement: issues in constructed response, performance testing, and portfolio assessment
159 Citations1993
Phi Delta KappanThe Computer Moves into Essay Grading: Updating the Ancient Test.
150 Citations1995Ellis B. Page, Nancy S. Petersen
A blind test has demonstrated that a computer can simulate the judgment of a group of human judges on a brand-new set of essays, and the bottom line is that the computer program did predict human judgments well - perhaps even better than three human judges.
Educational Measurement Issues and PracticeValidity and Automad Scoring: It's Not Only the Scoring
143 Citations1998Randy Elliot Bennett, Isaac I. Bejar
This research investigates the nature of the interplay among construct definition, task design, examinee interface, tutorial, test development tools, and automated scoring and reporting in the context of knowledge representation systems.
Computers in Human BehaviorStumping e-rater:challenging the validity of automated essay scoring
140 Citations2002Donald E. Powers, Jill Burstein +3 more
Information is provided on ways in which e-rater, and perhaps other automated essay scoring systems, may fail to provide accurate evaluations, if used as the sole method of scoring in high-stakes assessments.
Applied Measurement in EducationA Review of Strategies for Validating Computer-Automated Scoring
128 Citations2002Yongwei Yang, Chad W. Buckendahl +2 more
A conceptual framework and general recommendations for designing validation studies for CAS procedures are presented and the current practice of validating CAS-system-generated scores is reviewed.
An unsupervised method for detecting grammatical errors
127 Citations2000Martin Chodorow, Claudia Leacock
An unsupervised method for detecting grammatical errors by inferring negative evidence from edited textual corpora and the errorrecognition system, ALEK, performs with about 80% precision and 20% recall.
Criterion SM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays
123 Citations2003Jill Burstein, Martin Chodorow +1 more
The Criterion SM Online Essay Evaluation Service is described, a web-based system that provides automated scoring and evaluation of student essays that outperform baseline algorithms and some of the tools agree with human judges as often as two judges agree with each other.
Natural Language EngineeringIdentifying off-topic student essays without topic-specific training data
79 Citations2006Derrick Higgins, Jill Burstein +1 more
An algorithm is described that detects when a student's essay is off-topic without requiring a set of topic-specific essays for training, and provides more detailed information about the way in which an essay diverges from the requested essay topic.
ETS Research Report SeriesVALIDITY AND AUTOMATED SCORING: IT'S NOT ONLY THE SCORING
68 Citations1997Randy Elliot Bennett, Isaac I. Bejar
It is argued that a comprehensive discussion of validity and automated scoring includes the interplay among construct definition and test and task design; examinee interface; tutorial; test development tools; automated scoring; and reporting and that the interdependency among computer-based test components provides a unique opportunity to greatly improve educational and occupational assessment.
Journal of Educational Computing ResearchComparing the Validity of Automated and Human Scoring of Essays
46 Citations2002Donald E. Powers, Jill Burstein +3 more
This study examined the relationship of scores from a graduate level writing assessment to several independent, non-test indicators of examinees' writing skills—both for automated scores and for scores assigned by trained human readers.
ETS Research Report SeriesTHE COLLEGE BOARD VOCABULARY STUDY
31 Citations1994Hunter M. Breland, Robert J. Jones +4 more
ETS Research Report SeriesNEW SAT® WRITING PROMPT STUDY: ANALYSES OF GROUP IMPACT AND RELIABILITY
14 Citations2004Hunter M. Breland, Melvin Kubota +3 more
Results of the impact analyses revealed no significant prompt type effects for ethnic, gender, or language groups, although there were significant differences in mean scores for ethnic and gender groups for all prompts.
ETS Research Report SeriesSTATISTICAL AND MEASUREMENT PROPERTIES OF FEATURES USED IN ESSAY ASSESSMENT
6 Citations2004Shelby J. Haberman