Learning Ensembles from Bites: A Scalable and Accurate Approach
Journal of Machine Learning ResearchPublished 1 December 2004
Nitesh V. Chawla, Lawrence Hall, Kevin W. Bowyer, W. Philip Kegelmeyer
Citations122
SJR quartileQ1
SJR score2.02
SNIP3.07
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Voting many classifiers built on small subsets of data ("pasting small votes") is a promising approach for learning from massive data sets, one that can utilize the power of boosting and bagging.
Abstract
Bagging and boosting are two popular ensemble methods that typically achieve better accuracy than a single classifier. These techniques have limitations on massive data sets, because the size of the data set can be a bottleneck. Voting many classifiers built on small subsets of data (pasting small votes) is a promising approach for learning from massive data sets, one that can utilize the power of boosting and bagging. We propose a framework for building hundreds or thousands of such classifiers on small subsets of data in a distributed environment. Experiments show this approach is fast, accurate, and scalable.
Keywords
Computer Science
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Nucleic Acids ResearchThe Protein Data Bank
39,839 Citations2000Helen M. Berman
The goals of the PDB are described, the systems in place for data deposition and access, how to obtain further information, and near-term plans for the future development of the resource are described.
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
IEEE Transactions on Pattern Analysis and Machine IntelligenceThe random subspace method for constructing decision forests
6,737 Citations1998Tin Kam Ho
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
IEEE ExpertData mining and knowledge discovery: making sense out of data
4,643 Citations1996U.M. Feyyad
Find loads of the data mining and knowledge discovery making sense out of data book catalogues in this site as the choice of you visiting this page.
Machine LearningAn Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization
2,950 Citations2000Thomas G. Dietterich
The experiments show that in situations with little or no classification noise, randomization is competitive with (and perhaps slightly superior to) bagging but not as accurate as boosting, and sometimes better than randomization.
Machine LearningAn Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants
2,644 Citations1999Eric Bauer, Ron Kohavi
It is found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit, and that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)The Cascade-Correlation learning architecture
2,401 Citations2018Scott E. Fahlman, Christian Lebière
The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network.
Machine LearningMeasures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
2,288 Citations2003Ludmila I. Kuncheva, Christopher J. Whitaker
Although there are proven connections between diversity and accuracy in some special cases, the results raise some doubts about the usefulness of diversity measures in building classifier ensembles in real-life pattern recognition problems.
A streaming ensemble algorithm (SEA) for large-scale classification
1,199 Citations2001W. Nick Street, YongSeog Kim
A fast algorithm for large-scale or streaming data that classifies as well as a single decision tree built on all the data, requires approximately constant memory, and adjusts quickly to concept drift is presented.
Efficient progressive sampling
353 Citations1999Foster Provost, David Jensen +1 more
This work analyzes methods for progressive sampling using progressively larger samples as long as model accuracy improves, and concludes that progressive sampling can be remarkably efficient.
Machine LearningPasting Small Votes for Classification in Large Databases and On-Line
307 Citations1999Leo Breiman
This paper studies one class of methods which give accuracy comparable to that which could have be obtained if all data could have been held in core and which are computationally fast.
Data Mining and Knowledge DiscoveryA Survey of Methods for Scaling Up Inductive Algorithms
278 Citations1999Foster Provost, Venkateswarlu Kolluri
This paper summarizes, categorizes, and compares existing work on scaling up inductive algorithms, focusing on algorithms that build decision trees and rule sets, in order to provide focus and specific details.
Pattern Recognition LettersAn approach to the automatic design of multiple classifier systems
197 Citations2001Giorgio Giacinto, Fabio Roli
Results on the classification of multisensor remote sensing images show that an approach to the automatic design of multiple classifier systems aimed at selecting the subset made up of the most accurate and diverse classifiers allows the design of effective multiple classifiers.
Is independence good for combining classifiers?
196 Citations2002Ludmila I. Kuncheva, Christopher J. Whitaker +2 more
It is shown that dependent classifiers could offer a dramatic improvement over the individual accuracy, however, the relationship between dependency and accuracy of the pool is ambivalent.
Lecture notes in computer scienceLimiting the Number of Trees in Random Forests
124 Citations2001Patrice Latinne, Olivier Debeir +1 more
A simple procedure that a priori determines a minimum number of classifiers to combine in order to obtain a prediction accuracy level similar to the one obtained with the combination of larger ensembles based on the McNemar non-parametric test of significance is proposed.
Applied Physics Letters10.1162/153244304322972694
119 Citations2000
A large-scale experimental comparison of logistic regression and tree induction is presented, assessing classification accuracy and the quality of rankings based on class-membership probabilities, and a learning-curve analysis is used to examine the relationship of these measures to the size of the training set.
Scaling up: distributed machine learning with cooperation
110 Citations1996Foster Provost, Daniel N. Hennessy
This paper investigates the use of distributed processing to take advantage of the often dormant PCs and workstations available on local networks and demonstrates the power of the method by learning from a massive data set taken from the domain of cellular fraud detection.
Distributed and Parallel DatabasesBoosting Algorithms for Parallel and Distributed Learning
91 Citations2002Aleksandar Lazarević, Zoran Obradović
A framework for parallel and distributed boosting algorithms intended for efficient integrating specialized classifiers learned over very large, distributed and possibly heterogeneous databases that cannot fit into main computer memory is proposed.
Lecture notes in computer scienceA New Ensemble Diversity Measure Applied to Thinning Ensembles
86 Citations2003Robert E. Banfield, Lawrence Hall +2 more
A new way of describing the diversity of an ensemble of classifiers, the Percentage Correct Diversity Measure, is introduced and it is shown that diversity is generally modeled by the measure and ensembles can be made smaller without loss in accuracy.
Pattern Recognition LettersDistributed learning with bagging-like performance
56 Citations2002Nitesh V. Chawla, Thomas E. Moore +4 more
The results indicate that the simple approach of creating a committee of n classifiers from disjoint partitions each of size 1/n in a distributed way results in a classifier which has a bagging-like performance gain.
Elsevier eBooksDecision Theoretic Subsampling for Induction on Large Databases
52 Citations1993Ron Musick, Jason Catlett +1 more
This paper addresses the questions of assessing when the choice may be made with a given expected error, and determining a sampling strategy that minimizes the computation cost of making it, by using a subsample for this calculation.
Creating ensembles of classifiers
46 Citations2002Nitesh V. Chawla, Steven A. Eschrich +1 more
The significance of the finding is that a partition strategy for even small/moderate sized datasets when combined with bagging can yield better performance than applying a single learner using the entire dataset.
Lecture notes in computer scienceDistributed Pasting of Small Votes
26 Citations2002Nitesh V. Chawla, Lawrence Hall +3 more
A framework for building hundreds or thousands of such classifiers on small subsets of data in a distributed environment is proposed and experiments show this approach is fast, accurate, and scalable to massive datasets.
Lecture notes in computer scienceLearning Rules from Distributed Data
20 Citations2000Lawrence Hall, Nitesh V. Chawla +2 more
It is shown that this approach compares to directly creating a rule set from the aggregate data and promises faster learning, and an approach has been developed to extend the degree of parallelism achieved before this problem takes over.
ComputerData mining for the corporate masses?
18 Citations2002N. Leavitt
The paper considers several key data mining trends that have emerged and discusses standards for various aspects of data mining, which may be changing because of new techniques and technologies.
AVATAR -- Adaptive Visualization Aid for Touring And Recovery
3 Citations2000Lawrence Hall, Kevin W. Bowyer +3 more
A report on the initial development of software which uses a standard visualization tool to determine, label and display salient regions in large 3D physics simulation datasets, called AVATAR (Adaptive Visualization Aid for Touring and Recovery).
系统仿真学报Learning to Predict in Complex Biological Domains
2 Citations2002StevenEschrich
This paper describes an ensemble approach using subsampling that scales well with dataset size and shows that both protein secondary structure prediction and drug discovery prediction can be improved by the use of over-generalization, specifically through theUse of ensembles of small subsamples.