Toward Optimal Active Learning through Sampling Estimation of Error Reduction
Published 28 June 2001
Nicholas Roy, Andrew McCallum
Citations824
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
This paper presents an active learning method that di-rectly optimizes expected future error. This is in con-trast to many other popular techniques that instead aim to reduce version space size. These other meth-ods are popular because for many learning models, closed form calculation of the expected future error is intractable. Our approach is made feasible by taking a sampling approach to estimating the expected reduc-tion in error due to the labeling of a query. In exper-imental results on two real-world data sets we reach high accuracy very quickly, sometimes with four times fewer labeled examples than competing methods. 1.
Keywords
Computer Science
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
IEEE Transactions on Information TheoryDivergence measures based on the Shannon entropy
5,027 Citations1991Jinfeng Lin
A novel class of information-theoretic divergence measures based on the Shannon entropy is introduced, which do not require the condition of absolute continuity to be satisfied by the probability distributions involved and are established in terms of bounds.
A comparison of event models for naive bayes text classification
3,224 Citations1998Andrew McCallum, Kamal Nigam
It is found that the multi-variate Bernoulli performs well with small vocabulary sizes, but that the multinomial performs usually performs even better at larger vocabulary sizes--providing on average a 27% reduction in error over the multi -variateBernoulli model at any vocabulary size.
Query by committee
1,657 Citations1992H. Sebastian Seung, Manfred Opper +1 more
It is suggested that asymptotically finite information gain may be an important characteristic of good query algorithms, in which a committee of students is trained on the same data set.
Incremental and Decremental Support Vector Machine Learning
1,158 Citations2000Gert Cauwenberghs, Tomaso Poggio
An on-line recursive algorithm for training support vector machines, one vector at a time, is presented and interpretation of decremental unlearning in feature space sheds light on the relationship between generalization and geometry of the data.
Machine LearningSelective Sampling Using the Query by Committee Algorithm
1,118 Citations1997Yoav Freund, H. Sebastian Seung +2 more
It is shown that if the two-member committee algorithm achieves information gain with positive lower bound, then the prediction error decreases exponentially with the number of queries, and this exponential decrease holds for query learning of perceptrons.
Distributional clustering of English words
994 Citations1993Fernando Pereira, Naftali Tishby +1 more
Applied Physics Letters10.1162/153244302760185243
898 Citations2000
Experimental results showing that employing the active learning method can significantly reduce the need for labeled training instances in both the standard inductive and transductive settings are presented.
Using Maximum Entropy for Text Classification
756 Citations1999Kamal Nigam, John Lafferty +1 more
This paper uses maximum entropy techniques for text classification by estimating the conditional distribution of the class variable given the document by comparing accuracy to naive Bayes and showing that maximum entropy is sometimes significantly better, but also sometimes worse.
Employing EM and Pool-Based Active Learning for Text Classification
681 Citations1998Andrew McCallum, Kamal Nigam
The Query-by-Committee method of active learning is modified to use the unlabeled pool for explicitly estimating document density when selecting examples for labeling, and the combination of EM and active learning requires only slightly more than half as many labeled training examples to achieve the same accuracy as either the improved active learning or EM alone.
Boosting in the limit: maximizing the margin of learned ensembles
238 Citations1998Adam J. Grove, Dale Schuurmans
The crucial question as to why boosting works so well in practice, and how to further improve upon it, remains mostly open, and it is concluded that no simple version of the minimum-margin story can be complete.
Journal of Artificial Intelligence ResearchCommittee-Based Sample Selection for Probabilistic Classifiers
150 Citations1999Shlomo Argamon-Engelson, Ido Dagan
A family of empirical methods for committee-based sample selection in probabilistic classification models, which evaluate the informativeness of an example by measuring the degree of disagreement between several model variants, are described.
Bayesian Averaging of Classifiers and the Overfitting Problem
145 Citations2000Pedro Domingos
Active Learning with Committees for Text Categorization
143 Citations1997Ray Liere, Prasad Tadepalli
This paper reports on experiments using a committee of Winnowbased learners and demonstrates that this approach can reduce the number of labeled training examples required over that used by a single Winnow learner by l-2 orders of magnitude.
Active Learning for Parameter Estimation in Bayesian Networks
134 Citations2000Simon Tong, Daphne Koller
This paper provides a theoretical framework for this problem, and an algorithm that chooses which active learning queries to generate based on the model learned so far, and presents experimental results showing that the active learning algorithm can significantly reduce the need for training data in many situations.
Selective sampling for nearest neighbor classifiers
23 Citations1999Michael Lindenbaum, Shaul Markovitch +1 more