Investigating the effects of selective sampling on the annotation task
Published 1 January 2005Open access
Ben Hachey, Beatrice Alex, Markus C. Becker
Citations51
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An active learning experiment for named entity recognition in the astronomy domain shows the utility of active learning, and inspects double annotation data from the same domain and quantifies potential problems concerning annotators' performance.
Abstract
We report on an active learning experiment for named entity recognition in the astronomy domain. Active learning has been shown to reduce the amount of labelled data required to train a supervised learner by selectively sampling more informative data points for human annotation. We inspect double annotation data from the same domain and quantify potential problems concerning annotators' performance. For data selectively sampled according to different selection metrics, we find lower inter-annotator agreement and higher per token annotation times. However, overall results confirm the utility of active learning.
Keywords
Computer Science
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
Query by committee
1,657 Citations1992H. Sebastian Seung, Manfred Opper +1 more
It is suggested that asymptotically finite information gain may be an important characteristic of good query algorithms, in which a committee of students is trained on the same data set.
arXiv (Cornell University)Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
1,574 Citations2002Erik F. Tjong Kim Sang
Journal of Artificial Intelligence ResearchActive Learning with Statistical Models
1,286 Citations1996David Cohn, Zoubin Ghahramani +1 more
This work shows how the same principles may be used to select data for two alternative, statistically-based learning architectures: mixtures of Gaussians and locally weighted regression.
Employing EM and Pool-Based Active Learning for Text Classification
681 Citations1998Andrew McCallum, Kamal Nigam
The Query-by-Committee method of active learning is modified to use the unlabeled pool for explicitly estimating document density when selecting examples for labeling, and the combination of EM and active learning requires only slightly more than half as many labeled training examples to achieve the same accuracy as either the improved active learning or EM alone.
Introduction to the bio-entity recognition task at JNLPBA
579 Citations2004Jin-Dong Kim, Tomoko Ohta +3 more
The JNLPBA shared task of bio-entity recognition using an extended version of the GENIA version 3 named entity corpus of MEDLINE abstracts is described and a general discussion of the approaches taken by participating systems is presented.
Computational LinguisticsThe reliability of a dialogue structure coding scheme
401 Citations1997Jean Carletta, Stephen Isard +4 more
This paper describes the reliability of a dialogue structure coding scheme based on utterance function, game structure, and higher-level transaction structure that has been applied to a corpus of spontaneous task-oriented spoken dialogues.
Active Learning for Natural Language Parsing and Information Extraction
290 Citations1999Cynthia A. Thompson, Mary Elaine Califf +1 more
It is shown that active learning can signicantly reduce the number of annotated examples required to achieve a given level of performance for these complex tasks: semantic parsing and information extraction.
Active Learning with Statistical Models.
221 Citations1995David Cohn, Zoubin Ghahramani +1 more
Named entity recognition with character-level models
195 Citations2003Dan Klein, Joseph Smarr +2 more
Two named-entity recognition models which use characters and character n-grams either exclusively or as an important part of their data representation are discussed, both of which are a character-level HMM with minimal context information and a maximum-entropy conditional markov model with substantially richer context features.
Journal of Artificial Intelligence ResearchCommittee-Based Sample Selection for Probabilistic Classifiers
150 Citations1999Shlomo Argamon-Engelson, Ido Dagan
A family of empirical methods for committee-based sample selection in probabilistic classification models, which evaluate the informativeness of an example by measuring the degree of disagreement between several model variants, are described.
Inter-annotator Agreement for a German Newspaper Corpus
123 Citations2000Thorsten Brants
This paper presents the results of an investigation on inter-annotator agreement for the NEGRA corpus, consisting of German newspaper texts, and identifies categories causing large numbers of differences and categories that are handled inconsistently.
BMC BioinformaticsExploring the boundaries: gene and protein identification in biomedical text
108 Citations2005Jenny Rose Finkel, Shipra Dingare +4 more
Central contributions are rich use of features derived from the training data at multiple levels of granularity, a focus on correctly identifying entity boundaries, and the innovative use of several external knowledge sources including full MEDLINE abstracts and web searches.
Rule writing or annotation
82 Citations2000Grace Ngai, David Yarowsky
Results show that it is more efficient and more successful by several measures to train a system using active learning annotation rather than hand-crafted rule writing at a comparable level of human labor investment.
Ensemble-based Active Learning for Parse Selection.
49 Citations2004Miles Osborne, Jason Baldridge
This work shows that at times, an ensemble model trained with randomly sampled examples can outperform a single model trained using AL, and shows that even this much stronger baseline model can be improved upon.
Optimising Selective Sampling for Bootstrapping Named Entity Recognition
31 Citations2005Markus C. Becker, Benjamin Clayton Hachey +2 more
Several committee-based metrics for quantifying the disagreement between classifiers built using multiple views are evaluated, and it is demonstrated that the choice of metric can be optimised in simulation experiments with existing annotated data from different domains.
ArXiv.orgRule Writing or Annotation: Cost-efficient Resource Usage for Base Noun Phrase Chunking
18 Citations2001Grace Ngai, David Yarowsky
Multilingual resources for entity extraction
17 Citations2003Stephanie Strassel, Alexis Mitchell +1 more
The scope of annotation and research tasks within these programs, some of the challenges of multilingual corpus development for entity extraction, are described, and a description of the corpora developed to support this research are described.