Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk
Published 5 June 2010
Joel Tetreault, Elena Filatova, Martin Chodorow
Citations28
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Results from two pilot studies are presented which show that using the Amazon Mechanical Turk for preposition error annotation is as effective as using trained raters, but at a fraction of the time and cost.
Abstract
hunter.cuny.edu In this paper we present results from two pilot studies which show that using the Amazon Mechanical Turk for preposition error annotation is as effective as using trained raters, but at a fraction of the time and cost. Based on these results, we propose a new evaluation method which makes it feasible to compare two error detection systems tested on different learner data sets. 1
Keywords
Computer Science
Cheap and fast---but is it good?
1,906 Citations2008Rion Snow, Brendan O’Connor +2 more
This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
Get another label? improving data quality and data mining using multiple, noisy labelers
1,114 Citations2008Victor S. Sheng, Foster Provost +1 more
The results show clearly that when labeling is not perfect, selective acquisition of multiple labels is a strategy that data miners should have in their repertoire; for certain label-quality/cost regimes, the benefit is substantial.
Fast, cheap, and creative
426 Citations2009Chris Callison-Burch
It is found that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does, Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.
Natural Language EngineeringDetecting errors in English article usage by non-native speakers
224 Citations2006Na-Rae Han, Martin Chodorow +1 more
A maximum entropy classifier was trained to select among a/an, the, or zero article for noun phrases (NPs), based on a set of features extracted from the local context of each, and used to detect article errors in TOEFL essays of native speakers of Chinese, Japanese, and Russian.
The ups and downs of preposition error detection in ESL writing
174 Citations2008Joel Tetreault, Martin Chodorow
A methodology for detecting preposition errors in the writing of non-native English speakers and addressing the problem of annotation and evaluation by showing how current approaches of using only one rater can skew system evaluation.
Using Contextual Speller Techniques and Language Modeling for ESL Error Correction
140 Citations2008Michael Gamon, Jianfeng Gao +5 more
A decisiontree approach inspired by contextual spelling systems for detection and correction suggestions, and a large language model trained on the Gigaword corpus to provide additional information to filter out spurious suggestions are used.
Native judgments of non-native usage
66 Citations2008Joel Tetreault, Martin Chodorow
It is shown how using only one rater can skew system evaluation and then a sampling approach is presented that makes it possible to evaluate a system more efficiently.
CALICO JournalAutomatic Detection of Preposition Errors in Learner Writing
43 Citations2009Rachele De Felice, Stephen Pulman
An approach to the automatic correction of preposition errors in L2 English, based on a maximum entropy classifier, achieves average precision and recall of 35% on this task.