Classifier Ensembles with a Random Linear Oracle
IEEE Transactions on Knowledge and Data EngineeringPublished 14 March 2007
Ludmila I. Kuncheva, Juan J. Rodríguez
Citations155
SJR quartileQ1
SJR score2.57
SNIP3.30
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A combined fusion-selection approach to classifier ensemble design where each classifier in the ensemble is replaced by a miniensemble of a pair of subclassifiers with a random linear oracle to choose between the two.
Abstract
We propose a combined fusion-selection approach to classifier ensemble design. Each classifier in the ensemble is replaced by a miniensemble of a pair of subclassifiers with a random linear oracle to choose between the two. It is argued that this approach encourages extra diversity in the ensemble while allowing for high accuracy of the individual ensemble members. Experiments were carried out with 35 data sets from UCI and 11 ensemble models. Each ensemble model was examined with and without the oracle. The results showed that all ensemble methods benefited from the new approach, most markedly so random subspace and bagging. A further experiment with seven real medical data sets demonstrates the validity of these findings outside the UCI data collection
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Elsevier eBooksData Mining: Practical Machine Learning Tools and Techniques
25,718 Citations2011Ian H. Witten, Eibe Frank
Journal of Computer and System SciencesA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
20,320 Citations1997Yoav Freund, Robert E. Schapire
Wiley series in probability and statisticsStatistical Methods for Rates and Proportions
18,048 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
Machine LearningBagging predictors
16,377 Citations1996Leo Breiman
Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy.
Statistical Comparisons of Classifiers over Multiple Data Sets
11,198 Citations2006Janez Demšar
A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
IEEE Transactions on Pattern Analysis and Machine IntelligenceThe random subspace method for constructing decision forests
6,737 Citations1998Tin Kam Ho
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
Neural ComputationAdaptive Mixtures of Local Experts
4,988 Citations1991Robert A. Jacobs, Michael I. Jordan +2 more
A new supervised learning procedure for systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases, which is demonstrated to be able to be solved by a very simple expert network.
TechnometricsCombining Pattern Classifiers: Methods and Algorithms
3,234 Citations2005Subhash Bagui
This chapter discusses the development of the Spatial Point Pattern Analysis Code in S–PLUS, which was developed in 1993 by P. J. Diggle and D. C. Griffith.
Machine LearningMeasures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
2,288 Citations2003Ludmila I. Kuncheva, Christopher J. Whitaker
Although there are proven connections between diversity and accuracy in some special cases, the results raise some doubts about the usefulness of diversity measures in building classifier ensembles in real-life pattern recognition problems.
IEEE Transactions on Pattern Analysis and Machine IntelligenceRotation Forest: A New Classifier Ensemble Method
1,877 Citations2006Juan J. Rodríguez, Ludmila I. Kuncheva +1 more
This work examined the rotation forest ensemble on a random selection of 33 benchmark data sets from the UCI repository and compared it with bagging, AdaBoost, and random forest and prompted an investigation into diversity-accuracy landscape of the ensemble models.
IEEE Transactions on Pattern Analysis and Machine IntelligenceCombination of multiple classifiers using local accuracy estimates
949 Citations1997K. Woods, W. Philip Kegelmeyer +1 more
Journal of Machine Learning ResearchStatistical Comparisons of Classifiers over Multiple Data Sets
875 Citations2006DemšarJanez
Machine LearningMultiBoosting: A Technique for Combining Boosting and Wagging
699 Citations2000Geoffrey I. Webb
MultiBoosting is an extension to the highly successful AdaBoost technique for forming decision committees that is able to harness both AdaBoost's high bias and variance reduction with wagging's superior variance reduction.
Journal of Artificial Intelligence ResearchA System for Induction of Oblique Decision Trees
583 Citations1994Sreerama K. Murthy, Simon Kasif +1 more
This system, OC1, combines deterministic hill-climbing with two forms of randomization to find a good oblique split (in the form of a hyperplane) at each node of a decision tree.
Pruning Adaptive Boosting
512 Citations1997Dragos D. Margineantu, Thomas G. Dietterich
Machine LearningMultivariate Decision Trees
355 Citations1995Carla E. Brodley, Paul E. Utgoff
It is confirmed that allowing multivariate tests generally improves the accuracy of the resulting decision tree over a univariate tree, and several new methods for forming multivariate decision trees are presented.
IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)Switching between selection and fusion in combining classifiers: an experiment
351 Citations2002Ludmila I. Kuncheva
A combination of classifier selection and fusion by using statistical inference to switch between the two by offering a discussion on when to combine classifiers and how classifiers selection (static or dynamic) can be misled by the differences in the classifier team.
Proceedings of the IEEEA composite classifier system design: Concepts and methodology
332 Citations1979B.V. Dasarathy, B. V. Sheela
This study explores the scope for achieving enhanced recognition system performance through deployment of a composite classifier system consisting of two or more component classifiers which belong to different categories using a specific technique-Sequential Weight Increasing Factor Technique (SWIFT).
Information FusionCreating diversity in ensembles using artificial data
292 Citations2004Prem Melville, Raymond J. Mooney
A new method for generating ensembles, D ecorate (Diverse Ensemble Creation by Oppositional Relabeling of Artificial Training Examples), that directly constructs diverse hypotheses using additional artificially constructed training examples is presented.
Pattern Recognition LettersSoft combination of neural classifiers: A comparative study
238 Citations1999Antanas Verikas, Arūnas Lipnickas +3 more
This paper presents four schemes for soft fusion of the outputs of multiple classifiers using Zimmermann's compensatory operator, and an empirical evaluation substantiates the validity of the approaches with data-dependent weights, compared to various existing combination schemes of multipleclassifiers.
Machine LearningMultivariate decision trees
237 Citations1995Carla E. Brodley, Paul E. Utgoff
It is confirmed that allowing multivariate tests generally improves the accuracy of the resulting decision tree over a univariate tree, and several new methods for forming multivariate decision trees are presented and compared with several well-known methods.
Pattern Recognition LettersAn approach to the automatic design of multiple classifier systems
197 Citations2001Giorgio Giacinto, Fabio Roli
Results on the classification of multisensor remote sensing images show that an approach to the automatic design of multiple classifier systems aimed at selecting the subset made up of the most accurate and diverse classifiers allows the design of effective multiple classifiers.
Is independence good for combining classifiers?
196 Citations2002Ludmila I. Kuncheva, Christopher J. Whitaker +2 more
It is shown that dependent classifiers could offer a dramatic improvement over the individual accuracy, however, the relationship between dependency and accuracy of the pool is ambivalent.
IEEE Transactions on Evolutionary ComputationInducing oblique decision trees with evolutionary algorithms
167 Citations2003E. Canti-Paz, Chandrika Kamath
It is shown that the classification accuracy improves when the trees obtained with the EAs are combined in ensembles, and that sometimes it is possible to build the ensemble of evolutionary trees in less time than a single traditional oblique tree.
Induction of Oblique Decision Trees.
161 Citations1993David G. Heath, Simon Kasif +1 more
A randomized technique for partitioning examples using oblique hyperplanes to create smaller but equally accurate decision trees than those created by other methods, and is found to produce surprisingly small trees without losing predic-tive accuracy.
Pattern RecognitionA study on the performances of dynamic classifier selection based on local accuracy estimation
127 Citations2005Luca Didaci, Giorgio Giacinto +2 more
Upper bounds against which the performances of DCS by Local Accuracy estimation (DCS-LA) can be evaluated are proposed and the experimental results clearly show the effectiveness of the selection methods based on local accuracy estimates.
IEEE Transactions on Geoscience and Remote SensingMultiple classifier systems for supervised remote sensing image classification based on dynamic classifier selection
115 Citations2002P.C. Smits
Experiments show that the proposed method outperform MCS strategies based on belief functions and the DCS-LA in terms of minimum and maximum class accuracies and kappa coefficient of agreement and is a valid alternative to majority voting.
On growing better decision trees from data
112 Citations1996Steven L. Salzberg, Kolluru Venkata Sreerama Murthy
Two forms of data massaging, domain-independent and domain-specific, are distinguished and a new framework is outlined for the former, and the importance of the latter is illustrated in the context of two new, complex classification problems in astronomy.
Neural ComputationBoosted Mixture of Experts: An Ensemble Learning Scheme
101 Citations1999Ran Avnimelech, Nathan Intrator
A new supervised learning procedure for ensemble machines is presented, in which outputs of predictors, trained on different distributions, are combined by a dynamic classifier combination model.
IEEE Transactions on Neural NetworksOmnivariate decision trees
93 Citations2001Olcay Taner Yıldız, Ethem Alpaydın
This work proposes omnivariate trees where the decision node may be univariate, linear, or nonlinear depending on the outcome of comparative statistical tests on accuracy thus matching automatically the complexity of the node with the subproblem defined by the data reaching that node.
IEEE Transactions on Neural NetworksLocal linear perceptrons for classification
79 Citations1996Ethem Alpaydın, Michael I. Jordan
A structure composed of local linear perceptrons for approximating global class discriminants is investigated and it is concluded that even on such a high-dimensional problem, such local models are promising, much better than RBF's and use much less memory.
IEEE Transactions on Neural NetworksClassifiability-Based Omnivariate Decision Trees
62 Citations2005Yongjun Li, Minggang Dong +1 more
The proposed approach is fast and does not suffer from as high a computational burden as that incurred by typical model selection algorithms, and achieves better classification accuracy compared to statistical model select algorithms.
Lecture notes in computer scienceData Dependence in Combining Classifiers
57 Citations2003Mohamed S. Kamel, Nayer Wanas
A new categorization of combining schemes based on their dependence on the data patterns being classified is presented, arguing that data dependent, and especially explicitly datadependent, approaches represent the highest potential for improved performance.
IEEE Transactions on Systems Man and Cybernetics - Part A Systems and HumansMultivariate decision trees using linear discriminants and tabu search
49 Citations2003Xiao‐Bai Li, James R. Sweigart +4 more
The proposed algorithm appears to outperform some of the major classification algorithms in terms of classification accuracy, the proposed algorithm generates decision trees with relatively small sizes, and the proposed algorithms runs faster than most multivariate decision trees and its computing time increases linearly with data size, indicating that the algorithm is scalable to large datasets.
Lecture notes in computer scienceMML Inference of Oblique Decision Trees
42 Citations2004Peter Tan, David L. Dowe
The preliminary results show that on average and on most data-sets, MML oblique trees clearly perform better than both C4.5 and C5 on both “right”/“wrong” accuracy and probabilistic prediction – and with smaller trees, i.e., less leaf nodes.
IEEE Transactions on Neural NetworksThe pandemonium system of reflective agents
41 Citations1996F. J. Śmieja
It is shown how strongly the advantages gained from using a modularization technique depend on the nature of the problem and the superiority of the Pandemonium method over a single net on the first two test categories is contrasted with its limited advantages for the second two categories.
Pattern Recognition Letters‘Change-glasses’ approach in pattern recognition
40 Citations1993Ludmila I. Kuncheva
An idea is presented for on-line change of the recognition rule in dependence on the reliability of the classification decision, aiming at better accuracy.
Signal Processing Image CommunicationA dynamic classifier selection and combination approach to image region labelling
28 Citations2004Sameer Singh, Maneesha Singh
A new knowledge-based predictive approach based on estimating the Mahalanobis distance between test sample feature values and the corresponding probability distribution function from training data that selectively triggers classifiers is proposed.
Lecture notes in computer scienceDynamic Classifier Selection by Adaptive k-Nearest-Neighbourhood Rule
28 Citations2004Luca Didaci, Giorgio Giacinto
The use of neighbourhoods of adaptive shape and size are investigated to better cope with the difficulties of a reliable estimation of local accuracies and show that performance improvements can be achieved by suitably tuning some additional parameters.
Pattern RecognitionSelected tree classifier combination based on both accuracy and error diversity
26 Citations2005Hong‐Jae Shin, S SOHN
Pattern RecognitionSelected tree classifier combination based on both accuracy and error diversity
26 Citations2004Hyejin Shin, So Young Sohn
Empirical evaluation using publicly available data sets confirms the superiority of the proposed approach over other classifier combining methods.
DynaBoost: Combining Boosted Hypotheses in a Dynamic Way
15 Citations1999Perry D. Moerland, Eddy Mayoraz
This work presents an extension of Freund and Schapire's AdaBoost algorithm that allows an input-dependent combination of the base hypotheses, and shows that the dynamic approach significantly improves the results on most data sets when (rather weak) perceptron base hypotheses are used.