Multi-Task Active Learning for Linguistic Annotations
Published 1 June 2008
Roi Reichart, Katrin Tomanek, Udo Hahn, Ari Rappoport
Citations94
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is shown that MTAL outperforms random selection and a stronger baseline, onesided example selection, in which one task is pursued using AL and the selected examples are provided also to the other task.
Abstract
We extend the classical single-task active learning (AL) approach. In the multi-task active learning (MTAL) paradigm, we select examples for several annotation tasks rather than for a single one as usually done in the context of AL. We introduce two MTAL metaprotocols, alternating selection and rank combination, and propose a method to implement them in practice. We experiment with a twotask annotation scenario that includes named entity and syntactic parse tree annotations on three different corpora. MTAL outperforms random selection and a stronger baseline, onesided example selection, in which one task is pursued using AL and the selected examples are provided also to the other task. 1
Keywords
Computer Science
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Building a Large Annotated Corpus of English: The Penn Treebank
7,528 Citations1993Mitchell P. Marcus
Computational LinguisticsThe Proposition Bank: An Annotated Corpus of Semantic Roles
2,302 Citations2005Martha Palmer, Daniel Gildea +1 more
An automatic system for semantic role tagging trained on the corpus is described and the effect on its performance of various types of information is discussed, including a comparison of full syntactic parsing with a flat representation and the contribution of the empty trace categories of the treebank.
Computational LinguisticsHead-Driven Statistical Models for Natural Language Parsing
1,860 Citations2003Michael Collins
Three statistical models for natural language parsing are described, leading to approaches in which a parse tree is represented as the sequence of decisions corresponding to a head-centered, top-down derivation of the tree.
IEEE Transactions on Pattern Analysis and Machine IntelligenceDecision combination in multiple classifier systems
1,511 Citations1994Tin Kam Ho, J.J. Hull +1 more
This work proposes three methods based on the highest rank, the Borda count, and logistic regression for class set reranking that have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness.
Machine LearningSelective Sampling Using the Query by Committee Algorithm
1,118 Citations1997Yoav Freund, H. Sebastian Seung +2 more
It is shown that if the two-member committee algorithm achieves information gain with positive lower bound, then the prediction error decreases exponentially with the number of queries, and this exponential decrease holds for query learning of perceptrons.
Biomedical named entity recognition using conditional random fields and rich feature sets
527 Citations2004Burr Settles
As the wealth of biomedical knowledge in the form of literature increases, there is a rising need for effective natural language processing tools to assist in organizing, curating, and retrieving this information.
The GENIA corpus: an annotated research abstract corpus in molecular biology domain
259 Citations2002Tomoko Ohta, Yuka Tateisi +1 more
Multi-criteria-based active learning for named entity recognition
233 Citations2004Dan Shen, Jie Zhang +3 more
A multi-criteria-based active learning approach is proposed and effectively applied to named entity recognition and includes all the criteria using two selection strategies, both of which result in less labeling cost than single-criterion-based method.
Active learning for statistical natural language parsing
180 Citations2001Min Tang, Xiaoqiang Luo +1 more
This paper presents a method to reduce this demand using active learning, which selects what samples to annotate, instead of annotating blindly the whole training corpus, which shows that for about the same parsing accuracy, one only needs to annotates a third of the samples.
Active Learning for Word Sense Disambiguation with Methods for Addressing the Class Imbalance Problem
157 Citations2007Jingbo Zhu, Eduard Hovy
A bootstrap-based oversampling (BootOS) method is proposed that works better than ordinary over-sampling in active learning for word sense disambiguation and a prediction solution by considering max-confidence as the upper bound and min-error as the lower bound for stopping conditions is suggested.
Computational LinguisticsSample Selection for Statistical Parsing
135 Citations2004Rebecca Hwa
It is found that sample selection can significantly reduce the size of annotated training corpora and that uncertainty is a robust predictive criterion that can be easily applied to different learning models.
Empirical Methods in Natural Language ProcessingActive Learning and the Total Cost of Annotation.
106 Citations2004Jason Baldridge, Miles Osborne
This work explores one such strategy: using a model during annotation to automate some of the decisions during annotation, showing an 80% reduction in annotation cost compared with labeling randomly selected data with a single model.
An Approach to Text Corpus Construction which Cuts Annotation Costs and Maintains Reusability of Annotated Data
98 Citations2007Katrin Tomanek, Joachim Wermter +1 more
The issue whether a corpus annotated by means of AL can be re-used to train classifiers different from the ones employed by AL, supplying alternative feature sets as well is addressed.
Minimizing manual annotation cost in supervised training from corpora
84 Citations1996Sean P. Engelson, Ido Dagan
A family of methods for committee-based sample selection for probabilistic classifiers is described, and it is found that all variants achieve a significant reduction in annotation cost, though their computational efficiency differs.
Rule writing or annotation
82 Citations2000Grace Ngai, David Yarowsky
Results show that it is more efficient and more successful by several measures to train a system using active learning annotation rather than hand-crafted rule writing at a comparable level of human labor investment.
Investigating the effects of selective sampling on the annotation task
51 Citations2005Ben Hachey, Beatrice Alex +1 more
An active learning experiment for named entity recognition in the astronomy domain shows the utility of active learning, and inspects double annotation data from the same domain and quantifies potential problems concerning annotators' performance.
Lecture notes in computer scienceSense Annotation in the Penn Discourse Treebank
50 Citations2008Eleni Miltsakaki, Livio Robaldo +2 more
The rationale of the tagset, detailed descriptions of the senses with corpus examples, simple semantic definitions of each type of sense tags as well as informal descriptions ofThe inferences allowed at each level are provided.
A two-stage method for active learning of statistical grammars
34 Citations2005Markus C. Becker, Miles Osborne
A novel two-stage method is presented which first targets sentences which cannot be reliably selected using uncertainty sampling, and then applies standard uncertainty sampling to the remaining sentences and performs better than pure uncertainty sampling and better than an ensemble method based on bagged ensemble members only.
An Ensemble Method for Selection of High Quality Parses
28 Citations2007Roi Reichart, Ari Rappoport
A Sample Ensemble Parse Assessment (SEPA) algorithm for detecting parse quality that uses a function of the agreement among several copies of a parser, each of which trained on a different sample from the training data, to assess parse quality.
Approximating Learning Curves for Active-Learning-Driven Annotation
13 Citations2008Katrin Tomanek, Udo Hahn
A method for committee-based AL to approximate the progression of the learning curve based on the disagreement among the committee members is proposed, which relies on a separate, unlabeled corpus and is thus well suited for situations where a labeled gold standard is not available or would be too expensive to obtain.