Ensemble diversity measures and their application to thinning
Information FusionPublished 29 May 2004
Robert E. Banfield, Lawrence Hall, Kevin W. Bowyer, W. Philip Kegelmeyer
Citations216
SJR quartileQ1
SJR score4.13
SNIP4.60
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The percentage correct diversity measure shows a greatest correlation between the increase in voted ensemble accuracy and the diversity value, and the methods proposed for thinning show that ensembles can be made smaller without loss in accuracy.
Abstract
The diversity of an ensemble of classifiers can be calculated in a variety of ways. Here a diversity metric and a means for altering the diversity of an ensemble, called “thinning”, are introduced. We evaluate thinning algorithms created by several techniques on 22 publicly available datasets. When compared to other methods, our percentage correct diversity measure shows a greatest correlation between the increase in voted ensemble accuracy and the diversity value. Also, the analysis of different ensemble creation methods indicates that they generate different levels of diversity. Finally, the methods proposed for thinning show that ensembles can be made smaller without loss in accuracy.
Keywords
Computer Science
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
Machine LearningBagging predictors
16,377 Citations1996Leo Breiman
Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy.
Lecture notes in computer scienceEnsemble Methods in Machine Learning
7,862 Citations2000Thomas G. Dietterich
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
IEEE Transactions on Pattern Analysis and Machine IntelligenceThe random subspace method for constructing decision forests
6,737 Citations1998Tin Kam Ho
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
Programs for Machine Learning
5,794 Citations1994Steven L. Salzberg, Alberto M. Segre
In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.
Random decision forests
4,947 Citations2002Tin Kam Ho
IEEE Transactions on Pattern Analysis and Machine IntelligenceNeural network ensembles
4,269 Citations1990Lars Kai Hansen, Peter Salamon
It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks, which helps improve the performance and training of neural networks for classification.
Machine LearningAn Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization
2,950 Citations2000Thomas G. Dietterich
The experiments show that in situations with little or no classification noise, randomization is competitive with (and perhaps slightly superior to) bagging but not as accurate as boosting, and sometimes better than randomization.
Machine LearningMeasures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
2,288 Citations2003Ludmila I. Kuncheva, Christopher J. Whitaker
Although there are proven connections between diversity and accuracy in some special cases, the results raise some doubts about the usefulness of diversity measures in building classifier ensembles in real-life pattern recognition problems.
Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical CharacterVII. On the association of attributes in statistics: with illustrations from the material of the childhood society, &c
400 Citations1900George Yule
In the ordinary theory of statistical correlation, normal or otherwise, material susceptible of continuous variation, or at least of variation by a considerable number of discontinuous steps, certain practical cases arise, where either no variation is thinkable at all, or else is not measured or possibly measurable.
Machine LearningPasting Small Votes for Classification in Large Databases and On-Line
307 Citations1999Leo Breiman
This paper studies one class of methods which give accuracy comparable to that which could have be obtained if all data could have been held in core and which are computationally fast.
Pattern Recognition LettersAn approach to the automatic design of multiple classifier systems
197 Citations2001Giorgio Giacinto, Fabio Roli
Results on the classification of multisensor remote sensing images show that an approach to the automatic design of multiple classifier systems aimed at selecting the subset made up of the most accurate and diverse classifiers allows the design of effective multiple classifiers.
Is independence good for combining classifiers?
196 Citations2002Ludmila I. Kuncheva, Christopher J. Whitaker +2 more
It is shown that dependent classifiers could offer a dramatic improvement over the individual accuracy, however, the relationship between dependency and accuracy of the pool is ambivalent.
International Conference on Machine LearningA Unifeid Bias-Variance Decomposition and its Applications
172 Citations2000Pedro Domingos
A unified bias-variance decomposition that is applicable to squared loss, zero-one loss, variable misclassification costs, and other loss functions is presented and applied to decision tree learning, instancebased learning and boosting on a large suite of benchmark data sets.
Information FusionAn experimental study on diversity for bagging and boosting with linear classifiers
161 Citations2002Ludmila I. Kuncheva, Marina Skurichina +1 more
Diversity measures indicated that Boosting succeeds in inducing diversity even for stable classifiers whereas Bagging does not, confirming in a quantitative way the intuitive explanation behind the success of Boosting for linear classifiers for increasing training sizes, and the poor performance of Bagging.
Lecture notes in computer scienceLimiting the Number of Trees in Random Forests
124 Citations2001Patrice Latinne, Olivier Debeir +1 more
A simple procedure that a priori determines a minimum number of classifiers to combine in order to obtain a prediction accuracy level similar to the one obtained with the combination of larger ensembles based on the McNemar non-parametric test of significance is proposed.
Lecture notes in computer scienceThe “Test and Select” Approach to Ensemble Combination
92 Citations2000Amanda Sharkey, Noel Sharkey +2 more
The "test and select" methodolology for ensemble creation is considered, and the application of this methodology, and of ensembles in general, is explored further in two case studies.
Lecture notes in computer scienceA New Ensemble Diversity Measure Applied to Thinning Ensembles
86 Citations2003Robert E. Banfield, Lawrence Hall +2 more
A new way of describing the diversity of an ensemble of classifiers, the Percentage Correct Diversity Measure, is introduced and it is shown that diversity is generally modeled by the measure and ensembles can be made smaller without loss in accuracy.