Nonlinear Boosting Projections for Ensemble Construction
Universidad de Córdoba Insitutional Repository (Universidad de Córdoba)Published 1 May 2007Open access
Nicolás García‐Pedrajas, César García‐Osorio, Colin Fyfe
Citations90
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel approach for ensemble construction based on the use of nonlinear projections to achieve both accuracy and diversity of individual classifiers is proposed, which is less sensitive to noise in the data than boosting methods.
Abstract
In this paper we propose a novel approach for ensemble construction based on the use of nonlinear \nprojections to achieve both accuracy and diversity of individual classifiers. The proposed approach \ncombines the philosophy of boosting, putting more effort on difficult instances, with the basis of \nthe random subspace method. Our main contribution is that instead of using a random subspace, \nwe construct a projection taking into account the instances which have posed most difficulties to \nprevious classifiers. In this way, consecutive nonlinear projections are created by a neural network \ntrained using only incorrectly classified instances. The feature subspace induced by the hidden layer \nof this network is used as the input space to a new classifier. The method is compared with bagging \nand boosting techniques, showing an improved performance on a large set of 44 problems from the \nUCI Machine Learning Repository. An additional study showed that the proposed approach is less \nsensitive to noise in the data than boosting methods
Keywords
Computer Science
ACM Transactions on Intelligent Systems and TechnologyLIBSVM
41,340 Citations2011Chih-Chung Chang, Chih‐Jen Lin
Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.
Neural Networks: A Comprehensive Foundation
29,816 Citations1998Simon Haykin
Thorough, well-organized, and completely up to date, this book examines all the important aspects of this emerging technology, including the learning process, back-propagation learning, radial-basis function networks, self-organizing systems, modular networks, temporal processing and neurodynamics, and VLSI implementation of neural networks.
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
Cambridge University Press eBooksAn Introduction to Support Vector Machines and Other Kernel-based Learning Methods
13,883 Citations2000Nello Cristianini, John Shawe‐Taylor
Springer series in information sciencesSelf-Organizing Maps
10,377 Citations1997Teuvo Kohonen
The mathematical preliminaries, background, basic ideas, and implications of the Self-Organising Map algorithm are expounded in a manner which is accessible without prior expert knowledge.
Journal of Business and Economic StatisticsAn Introduction to Multivariate Statistical Analysis
9,232 Citations1986Robb J. Muirhead, T. W. Anderson
Lecture notes in computer scienceEnsemble Methods in Machine Learning
7,862 Citations2000Thomas G. Dietterich
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
The Annals of StatisticsAdditive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)
6,842 Citations2000Jerome H. Friedman, Trevor Hastie +1 more
This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.
IEEE Transactions on Pattern Analysis and Machine IntelligenceThe random subspace method for constructing decision forests
6,737 Citations1998Tin Kam Ho
A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity.
Programs for Machine Learning
5,794 Citations1994Steven L. Salzberg, Alberto M. Segre
In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.
ePrints Soton (University of Southampton)An Introduction to Support Vector Machines
4,120 Citations2000Nello Cristianini, John Shawe‐Taylor
Neural ComputationApproximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
3,654 Citations1998Thomas G. Dietterich
This article reviews five approximate statistical tests for determining whether one learning algorithm outperforms another on a particular learning task and measures the power (ability to detect algorithm differences when they do exist) of these tests.
Machine LearningAn Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization
2,950 Citations2000Thomas G. Dietterich
The experiments show that in situations with little or no classification noise, randomization is competitive with (and perhaps slightly superior to) bagging but not as accurate as boosting, and sometimes better than randomization.
Principal Components Analysis
2,664 Citations1989George H. Dunteman
Machine LearningAn Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants
2,644 Citations1999Eric Bauer, Ron Kohavi
It is found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit, and that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference.
Improved boosting algorithms using confidence-rated predictions
2,568 Citations1998Robert E. Schapire, Yoram Singer
The Annals of StatisticsBoosting the margin: a new explanation for the effectiveness of voting methods
2,313 Citations1998Peter L. Bartlett, Yoav Freund +2 more
It is shown that techniques used in the analysis of Vapnik's support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error.
Machine LearningMeasures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
2,288 Citations2003Ludmila I. Kuncheva, Christopher J. Whitaker
Although there are proven connections between diversity and accuracy in some special cases, the results raise some doubts about the usefulness of diversity measures in building classifier ensembles in real-life pattern recognition problems.
Machine LearningBoosTexter: A Boosting-based System for Text Categorization
2,194 Citations2000Robert E. Schapire, Yoram Singer
This work describes in detail an implementation, called BoosTexter, of the new boosting algorithms for text categorization tasks, and presents results comparing the performance of Boos Texter and a number of other text-categorization algorithms on a variety of tasks.
Artificial IntelligenceEnsembling neural networks: Many could be better than all
1,982 Citations2002Zhi‐Hua Zhou, Jianxin Wu +1 more
The bias-variance decomposition of the error is provided in this paper, which shows that the success of GASEN may lie in that it can significantly reduce the bias as well as the variance.
IEEE Transactions on Pattern Analysis and Machine IntelligenceRotation Forest: A New Classifier Ensemble Method
1,877 Citations2006Juan J. Rodríguez, Ludmila I. Kuncheva +1 more
This work examined the rotation forest ensemble on a random selection of 33 benchmark data sets from the UCI repository and compared it with bagging, AdaBoost, and random forest and prompted an investigation into diversity-accuracy landscape of the ensemble models.
Machine LearningStacked Regressions
1,129 Citations1996Leo Breiman
Stacking regressions is a method for forming linear combinations of different predictors to give improved prediction accuracy by using cross-validation data and least squares under non negativity constraints to determine the coefficients in the combination.
Machine LearningIs Combining Classifiers with Stacking Better than Selecting the Best One?
891 Citations2004Sašo Džeroski, Bernard Ženko
This work empirically evaluates several state-of-the-art methods for constructing ensembles of heterogeneous classifiers with stacking and shows that they perform (at best) comparably to selecting the best classifier from the ensemble by cross validation and proposes two extensions of this method using an extended set of meta-level features and multi-response model trees to learn at the meta- level.
Machine LearningMultiBoosting: A Technique for Combining Boosting and Wagging
699 Citations2000Geoffrey I. Webb
MultiBoosting is an extension to the highly successful AdaBoost technique for forming decision committees that is able to harness both AdaBoost's high bias and variance reduction with wagging's superior variance reduction.
Connection ScienceError Correlation and Error Reduction in Ensemble Classifiers
602 Citations1996Kagan Tumer, Joydeep Ghosh
This paper focuses on data selection and classifier training methods, in order to 'prepare' classifiers for combining, and discusses several methods that make the classifiers in an ensemble more complementary.
Neural ComputationPrediction Games and Arcing Algorithms
546 Citations1999Leo Breiman
The theory behind the success of adaptive reweighting and combining algorithms (arcing) such as Adaboost and others in reducing generalization error has not been well understood, and an explanation of whyAdaboost works in terms of its ability to produce generally high margins is offered.
Pruning Adaptive Boosting
512 Citations1997Dragos D. Margineantu, Thomas G. Dietterich
IEEE Transactions on Evolutionary ComputationEvolutionary ensembles with negative correlation learning
446 Citations2000Tetsuya Higuchi, Xin Yao +1 more
Experiments on two real-world problems demonstrate that EENCL can produce NN ensembles with good generalization ability.
Lecture notes in statisticsA Comparative Evaluation of Sequential Feature Selection Algorithms
446 Citations1996David W. Aha, Richard L. Bankert
Positive empirical results are reported on variants of sequential feature selection that might be more appropriate for some performance tasks, and it is argued for their serious consideration in similar learning tasks.
Neural ComputationCombined 5 × 2 cv F Test for Comparing Supervised Classification Learning Algorithms
428 Citations1999Ethem Alpaydm
A variant of the 5 2 cvt test is proposed that combines multiple statistics to get a more robust test, and simulation results show that this combined version of the test has lower type I error and higher power than5 2 cv proper.
Boosting with Early Stopping: Convergence and Consistency
326 Citations2003Tong Zhang, Bin Yu
Feature selection for ensembles
304 Citations1999David W. Opitz
This paper presents an ensemble feature selection approach that is based on genetic algorithms and shows improved performance over the popular and powerful ensemble approaches of AdaBoost and Bagging and demonstrates the utility of ensemble features selection.
Back Propagation is Sensitive to Initial Conditions
272 Citations1990John F. Kolen, Jordan Pollack
Wrappers for Performance Enhancements and Oblivious Decision Graphs.
254 Citations1995Ron Kohavi
This doctoral dissertation concludes that repeated runs of five-fold cross-validation give a good tradeoff between bias and variance for the problem of model selection used in later chapters.
Boosting in the limit: maximizing the margin of learned ensembles
238 Citations1998Adam J. Grove, Dale Schuurmans
The crucial question as to why boosting works so well in practice, and how to further improve upon it, remains mostly open, and it is concluded that no simple version of the minimum-margin story can be complete.
IEEE Transactions on Evolutionary ComputationCooperative Coevolution of Artificial Neural Network Ensembles for Pattern Classification
230 Citations2005Nicolás García‐Pedrajas, César Hervás‐Martínez +1 more
This paper proposes a general framework for designing neural network ensembles by means of cooperative coevolution, and applies the proposed model to ten real-world classification problems of a very different nature from the UCI machine learning repository and proben1 benchmark set.
Boosting as a Regularized Path to a Maximum Margin Classifier
228 Citations2004Saharon Rosset, Ji Zhu +1 more
It is built on recent work by Efron et al. to show that boosting approximately (and in some cases exactly) minimizes its loss criterion with an l1 constraint on the coefficient vector, and shows that as the constraint is relaxed the solution converges (in the separable case) to an "l1-optimal" separating hyper-plane.
Machine LearningUsing Correspondence Analysis to Combine Classifiers
210 Citations1999Christopher J. Merz
The method described uses the strategies of stacking and Correspondence Analysis to model the relationship between the learning examples and their classification by a collection of learned models, and a nearest neighbor method is applied within the resulting representation to classify previously unseen examples.
Lecture notes in computer scienceDiversity versus Quality in Classification Ensembles Based on Feature Selection
206 Citations2000Pádraig Cunningham, John G. Carney
It is argued that feature subset-selection has emerged as a useful technique for creating diversity in ensembles and this diversity needs to be monitored in the creation of the ensemble, and an entropy measure of the outputs of the Ensemble Diversity Index is proposed.
Lecture notes in computer scienceUsing Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error
182 Citations2001Gabriele Zenobi, Pádraig Cunningham
This paper presents a process for producing ensembles of classifiers based on different feature subsets that emphasizes diversity (ambiguity) in the ensemble members and finds that the ensemble based on ambiguity have lower generalization error.
MadaBoost: A Modification of AdaBoost
169 Citations2000Carlos Domingo, Osamu Watanabe
A new boosting algorithm MadaBoost is proposed that can be casted in the statistical query learning model [Kea93] and thus, it is robust to random classification noise [AL88].
Journal of Artificial Intelligence ResearchCIXL2: A Crossover Operator for Evolutionary Algorithms Based on Population Features
157 Citations2005Domingo Ortíz-Boyer, C. Hervás-Martínez +1 more
A crossover operator for evolutionary algorithms with real values that is based on the statistical theory of population distributions that takes into account the localization and dispersion features of the best individuals of the population with the objective that these features would be inherited by the offspring.
Machine LearningOnline Ensemble Learning: An Empirical Study
137 Citations2003Alan Fern, Robert Givan
IEEE Transactions on Pattern Analysis and Machine IntelligenceOn the algorithmic implementation of stochastic discrimination
132 Citations2000E. M. Kleinberg
An outline of the underlying mathematical theory of stochastic discrimination is outlined and a remark concerning boosting is made, which provides a theoretical justification for properties of that method observed in practice, including its ability to generalize.
Machine LearningImproved Generalization Through Explicit Optimization of Margins
131 Citations2000Llew Mason, Peter L. Bartlett +1 more
A theorem bounding the generalization performance of convex combinations in terms of general cost functions of the margin is proved, in contrast to previous results, which were stated in Terms of the particular cost function sgn(θ − margin).
International Journal of Pattern Recognition and Artificial IntelligenceMethods of Combining Multiple Classifiers with Different Features and Their Applications to Text-Independent Speaker Identification
121 Citations1997Ke Chen, Lan Wang +1 more
The simulations show that results yielded by the methods described in this paper are better than not only the individual classifiers' but also ones obtained by combining multiple classifiers with the same feature.
Face detection using mixtures of linear subspaces
111 Citations2002Ming–Hsuan Yang, N. Abuja +1 more
Two methods using mixtures of linear sub-spaces for face detection in gray level images using Kohonen's self-organizing map for clustering and Fisher linear discriminant to find the optimal projection for pattern classification are presented.
Option Decision Trees with Majority Votes
104 Citations1997Ron Kohavi, Clayton Kunz
The goal was to explore when option nodes are most useful and to control the growth of the trees so that additional complexity of little utility is limited and the results show that for the tested problems, the reduction in error rates can be achieved.
Pattern Recognition LettersA comparative study of neural network based feature extraction paradigms
87 Citations1999Boaz Lerner, Hugo Guterman +2 more
Tested on five real-world databases, the MLP provides the highest classification accuracy at the cost of deforming the data structure, whereas the linear models preserve the structure but usually with inferior accuracy.
Pattern classification using support vector machine ensemble
67 Citations2003Hyun‐Chul Kim, Shaoning Pang +3 more
Simulation results for handwritten digit recognition and fraud detection show that the proposed SVM ensemble with bagging or boosting greatly outperforms a single SVM in terms of classification accuracy.
Lecture notes in computer scienceBagging and the Random Subspace Method for Redundant Feature Spaces
63 Citations2001Marina Skurichina, Robert P. W. Duin
The effect of the redundancy in the data feature set on the performance of the random subspace method and bagging is studied and it appears that they are useful in different situations.
Multiclass Boosting for Weak Classifiers
61 Citations2005Günther Eibl, Karl-Peter Pfeiffer
Two alternative boosting algorithms which also minimize bounds on performance measures are proposed which are more easily applicable to already existing base classifiers and tend to converge faster than AdaBoost.M2.
Journal of Machine Learning ResearchStopping criterion for boosting based data reduction techniques: from binary to multiclass problem
33 Citations2003Marc Sebban, Richard Nock +1 more
Search strategies for ensemble feature selection in medical diagnostics
28 Citations2003Alexey Tsymbal, Pádraig Cunningham +2 more
This paper proposes two new sequential-search-based strategies for ensemble feature selection, and evaluates them, constructing ensembles of simple Bayesian classifiers for the problem of acute abdominal pain classification.
Comparing pure parallel ensemble creation techniques against bagging
26 Citations2004Lawrence Hall, Kevin W. Bowyer +4 more
Eight randomization-based approaches to creating an ensemble of decision-tree classifiers are evaluated, and it is found that none of them is consistently more accurate than standard bagging when tested for statistical significance.
Meta-learning orthographic and contextual models for language independent named entity recognition
23 Citations2003Robert Munro, Daren Ler +1 more
This paper presents a named entity classification system that utilises both orthographic and contextual information and the random subspace method was employed to generate and refine attribute models.
On boosting with polynomially bounded distributions
17 Citations2003Nader H. Bshouty, Dmitry Gavinsky
A framework is constructed which allows an algorithm to turn the distributions produced by some boosting algorithms into polynomially smooth distributions, with minimal performance loss, and demonstrates AdaBoost's application to the task of DNF learning using membership queries.
Lecture notes in computer scienceError Bounds for Aggressive and Conservative AdaBoost
11 Citations2003Ludmila I. Kuncheva
Three AdaBoost variants are distinguished based on the strategies applied to update the weights for each new ensemble member, and the aggressive variant and the less popular of the two conservative variants have lower error bounds than the classical AdaBoost.
Lecture notes in computer scienceA Method to Boost Support Vector Machines
7 Citations2002Lili Diao, Keyun Hu +2 more
This paper introduces an efficient way to boost SVM that embraces the idea of active learning to dynamically select "important" samples into training sample set for constructing base classifiers.
Neural Processing LettersEnsemble of Independent Factor Analyzers with Application to Natural Image Analysis
6 Citations2001Akio Utsugi
The ensemble of independent factor analyzers develop into feature detectors that resemble complex cells in mammalian visual systems in a more general framework of models, including overcomplete models allowing additive noise in the observables.