Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation
Published 6 June 2010Open access
Cem Akkaya, Alexander Conrad, Janyce Wiebe, Rada Mihalcea
Citations71
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Using MTurk to collect annotations for Subjectivity Word Sense Disambiguation (SWSD), a coarse-grained word sense disambigsuation task, is investigated, suggesting a greater role for MTurK with respect to constructing a large scale SWSD system in the future, promising substantial improvement in subjectivity and sentiment analysis.
Abstract
In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire good annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.
Keywords
Computer Science
Neural ComputationApproximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
3,654 Citations1998Thomas G. Dietterich
This article reviews five approximate statistical tests for determining whether one learning algorithm outperforms another on a particular learning task and measures the power (ability to detect algorithm differences when they do exist) of these tests.
Cheap and fast---but is it good?
1,906 Citations2008Rion Snow, Brendan O’Connor +2 more
This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
Computers and the HumanitiesAnnotating Expressions of Opinions and Emotions in Language
1,770 Citations2005Janyce Wiebe, Theresa Wilson +1 more
The manual annotation process and the results of an inter-annotator agreement study on a 10,000-sentence corpus of articles drawn from the world press are presented.
Utility data annotation with Amazon Mechanical Turk
625 Citations2008Alexander Sorokin, David Forsyth
This work shows how to outsource data annotation to Amazon Mechanical Turk, and describes results for several different annotation problems, including some strategies for determining when the task is well specified and properly priced.
Fast, cheap, and creative
426 Citations2009Chris Callison-Burch
It is found that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does, Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.
Word sense and subjectivity
296 Citations2006Janyce Wiebe, Rada Mihalcea
Empirical evidence is brought in support of the hypotheses that (1) subjectivity is a property that can be associated with word senses, and (2) word sense disambiguation can directly benefit from subjectivity annotations.
Data quality from crowdsourcing
252 Citations2009Pei-Yun Hsueh, Prem Melville +1 more
An empirical study is conducted to examine the effect of noisy annotations on the performance of sentiment classification models, and evaluate the utility of annotation selection on classification accuracy and efficiency.
Subjectivity word sense disambiguation
182 Citations2009Cem Akkaya, Janyce Wiebe +1 more
Empirical evidence is provided that SWSD is more feasible than full word sense disambiguation, and that it can be exploited to improve the performance of contextual subjectivity and sentiment analysis systems.
D-Scholarship@Pitt (University of Pittsburgh)Fine-grained subjectivity and sentiment analysis: recognizing the intensity, polarity, and attitudes of private states
137 Citations2008Theresa Wilson
This dissertation investigates the manual and automatic identification of linguistic expressions of private states in a corpus of news documents from the world press with results in automatic systems for performing fine-grained subjectivity analysis that significantly outperform baseline systems.
Getting Serious about Word Sense Disambiguation
77 Citations1997Hwee Tou Ng
It is argued that a large, human sensetagged corpus is also critical as well as necessary to achieve broad coverage, high accuracy word sense disambiguation, where the sense distinction is at the level of a good desk-top dictionary such as WORDNET.
Columbia Academic Commons (Columbia University)Inter-annotator agreement on a multilingual semantic annotation task
44 Citations2006Rebecca J. Passonneau, Nizar Habash +1 more
Inter-annotator agreement for the IAMTC project, which Parsed versions of English translations of news articles in Arabic, French, Hindi, Japanese, Korean and Spanish were annotated by up to ten annotators, is discussed.
Collecting a Why-Question Corpus for Development and Evaluation of an Automatic QA-System
29 Citations2008Joanna Mrozinski, Edward W. D. Whittaker +1 more
A Wikipedia-based corpus of Whyquestions and corresponding answers and articles is presented, built by a novel method: paid participants were contacted through a Web-interface, a procedure which allowed dynamic, fast and inexpensive development of data collection methods.