Robust Combining of Disparate Classifiers through Order Statistics
Pattern Analysis and ApplicationsPublished 7 June 2002
Kagan Tumer, Joydeep Ghosh
Citations33
SJR quartileQ2
SJR score0.56
SNIP1.03
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The trim and spread combiners are analysed, both based on linear combinations of the ordered classifier outputs, and it is shown that in the presence of uneven classifier performance, they often provide substantial gains over both linear and simple order statistics combiners.
Abstract
Integrating the outputs of multiple classifiers via combiners or meta-learners has led to substantial improvements in several difficult pattern recognition problems. In this article we investigate a family of combiners based on order statistics, for robust handling of situations where there are large discrepancies in performance of individual classifiers. Based on a mathematical modeling of how the decision boundaries are affected by order statistic combiners, we derive expressions for the reductions in error expected when simple output combination methods based on the the median, the maximum and in general, the ith order statistic, are used. Furthermore, we analyze the trim and spread combiners, both based on linear combinations of the ordered classifier outputs, and show that in the presence of uneven classifier performance, they often provide substantial gains over both linear and simple order statistics combiners. Experimental results on both real world data and standard public domain data sets corroborate these findings.
Keywords
Computer ScienceMathematics
Journal of Computer and System SciencesA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
20,320 Citations1997Yoav Freund, Robert E. Schapire
Lecture notes in computer scienceEnsemble Methods in Machine Learning
7,862 Citations2000Thomas G. Dietterich
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
Neural NetworksStacked generalization
7,478 Citations1992David H. Wolpert
The conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOn combining classifiers
5,227 Citations1998Josef Kittler, M. Hatef +2 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceNeural network ensembles
4,269 Citations1990Lars Kai Hansen, Peter Salamon
It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks, which helps improve the performance and training of neural networks for classification.
Neural ComputationNeural Networks and the Bias/Variance Dilemma
3,563 Citations1992Stuart Geman, Elie Bienenstock +1 more
It is suggested that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues.
Machine LearningThe Strength of Weak Learnability
3,293 Citations1990Robert E. Schapire
In this paper, a method is described for converting a weak learning algorithm into one that achieves arbitrarily high accuracy, and it is shown that these two notions of learnability are equivalent.
International Journal of Man-Machine StudiesSimplifying decision trees
2,502 Citations1987J. R. Quinlan
Techniques for simplifying decision trees while retaining their accuracy are discussed, described, illustrated, and compared on a test-bed of decision trees from a variety of domains.
The Annals of StatisticsBoosting the margin: a new explanation for the effectiveness of voting methods
2,313 Citations1998Peter L. Bartlett, Yoav Freund +2 more
It is shown that techniques used in the analysis of Vapnik's support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error.
IEEE Transactions on Systems Man and CyberneticsMethods of combining multiple classifiers and their applications to handwriting recognition
2,171 Citations1992Longchang Xu, Adam Krzyżak +1 more
On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly.
Neural Network Ensembles, Cross Validation, and Active Learning
1,799 Citations1994Anders Krogh, Jesper Vedelsby
It is shown how to estimate the optimal weights of the ensemble members using unlabeled data and how the ambiguity can be used to select new training data to be labeled in an active learning scheme.
IEEE Transactions on Pattern Analysis and Machine IntelligenceDecision combination in multiple classifier systems
1,511 Citations1994Tin Kam Ho, J.J. Hull +1 more
This work proposes three methods based on the highest rank, the Borda count, and logistic regression for class set reranking that have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness.
National Conference on Artificial IntelligenceBagging, boosting, and C4.S
1,262 Citations1996J. R. Quinlan
Machine LearningStacked Regressions
1,129 Citations1996Leo Breiman
Stacking regressions is a method for forming linear combinations of different predictors to give improved prediction accuracy by using cross-validation data and least squares under non negativity constraints to determine the coefficients in the combination.
Data Mining and Knowledge DiscoveryOn Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality
1,029 Citations1997Jerome H. Friedman
This work candramatically mitigate the effect of the bias associated with some simpleestimators like “naive” Bayes, and the bias induced by the curse-of-dimensionality on nearest-neighbor procedures.
Refining Initial Points for K-Means Clustering
999 Citations1998Paul S. Bradley, Usama M. Fayyad
A procedure for computing a refined starting condition from a given initial one that is based on an efficient technique for estimating the modes of a distribution that allows the iterative algorithm to converge to a “better” local minimum.
Neural ComputationNeural Network Classifiers Estimate Bayesian <i>a posteriori</i> Probabilities
993 Citations1991Michael D. Richard, Richard P. Lippmann
Results of Monte Carlo simulations performed using multilayer perceptron (MLP) networks trained with backpropagation, radial basis function (RBF) networks, and high-order polynomial networks graphically demonstrate that network outputs provide good estimates of Bayesian probabilities.
IEEE Transactions on Neural NetworksThe multilayer perceptron as an approximation to a Bayes optimal discriminant function
840 Citations1990D.W. Ruck, Steven K. Rogers +3 more
The multilayer perceptron, when trained as a classifier using backpropagation, is shown to approximate the Bayes optimal discriminant function.
World Scientific series in 20th century physicsWhen networks disagree: Ensemble methods for hybrid neural networks
764 Citations1995Michael Perrone, Leon N. Cooper
Experimental results show that the ensemble method dramatically improves neural network performance on difficult real-world optical character recognition tasks.
PubMed CentralUsing the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus
742 Citations1988Jack W. Smith, James E. Everhart +3 more
Testing the ability of an early neural network model, ADAP, to forecast the onset of diabetes mellitus in a high risk population of Pima Indians and comparing the results with those obtained from logistic regression and linear perceptron models using precisely the same training and forecasting sets.
Connection ScienceError Correlation and Error Reduction in Ensemble Classifiers
602 Citations1996Kagan Tumer, Joydeep Ghosh
This paper focuses on data selection and classifier training methods, in order to 'prepare' classifiers for combining, and discusses several methods that make the classifiers in an ensemble more complementary.
QUT ePrints (Queensland University of Technology)Boosting the margin: A new explanation for the effectiveness of voting methods
578 Citations1997Robert E. Schapire, Yoav Freund +2 more
Pruning Adaptive Boosting
512 Citations1997Dragos D. Margineantu, Thomas G. Dietterich
Neural NetworksCombining the results of several neural network classifiers
399 Citations1994Galina L. Rogova
This paper presents a combination method based on the Dempster–Shafer theory of evidence, which uses statistical information about the relative classification strengths of several classifiers to reduce misclassification error.
Elsevier eBooksError-Correcting Output Coding Corrects Bias and Variance
357 Citations1995Eun Bae Kong, Thomas G. Dietterich
An investigation of why the ECOC technique works, particularly when employed with decision-tree learning algorithms, shows that it can reduce the variance of the learning algorithm.
Neural ComputationBoosting and Other Ensemble Methods
355 Citations1994Harris Drucker, Corinna Cortes +3 more
A surprising result is shown for the original boosting algorithm: namely, that as the training set size increases, the training error decreases until it asymptotes to the test error rate.
CERN Document Server (European Organization for Nuclear Research)Learning with ensembles: How overfitting can be useful
340 Citations1995Peter Sollich, Anders Krogh
It is advantageous to use under-regularized students, which actually over-fit the training data, to improve ensemble generalization performance, in particular if the individual students are subject to noise in the training process.
Pattern RecognitionAnalysis of decision boundaries in linearly combined neural classifiers
331 Citations1996Kagan Tumer, Joydeep Ghosh
An analytical framework is provided to quantify the improvements in classification results due to combining and shows that combining networks linearly in output space reduces the variance of the actual decision region boundaries around the optimum boundary.
Neural NetworksDemocracy in neural nets: Voting schemes for classification
309 Citations1994Roberto Battiti, A.M. Colla
This paper found that the use of a small number of nets with a sufficiently large uncorrelation in their mistakes reaches a combined performance that is significantly higher than the best obtainable from the individual nets, with a negligible effort after starting from a pool of networks produced in the development phase of an application.
Journal of the American Statistical AssociationCombining Estimates in Regression and Classification
296 Citations1996Michael LeBlanc, Robert Tibshirani
A general framework for how to combine a collection of general regression fit vectors to obtain a better predictive model is developed and a cross-validation—based proposal called “model mix” or “stacking” is examined.
Generating Accurate and Diverse Members of a Neural-Network Ensemble
279 Citations1995David W. Opitz, Jude Shavlik
A technique called ADDEMUP is presented that uses genetic algorithms to directly search for an accurate and diverse set of trained networks and is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
Minds at UW (University of Wisconsin)Pattern Recognition Via Linear Programming: Theory and Application to Medical Diagnosis
257 Citations1989William H. Wolberg, O. L. Mangasarian +1 more
Lecture notes in computer scienceExperiments with Classifier Combining Rules
235 Citations2000Robert P. W. Duin, David M. J. Tax
It is shown that there is no overall winning combining rule and that bad classifiers as well as bad feature sets may contain valuable information for performance improvement by combining rules.
Repository KITopen (Karlsruhe Institute of Technology)PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms
234 Citations1994Lutz Prechelt
The purpose of the problem and rule collection is to give researchers easy access to data for the evaluation of their algorithms and networks and to make direct comparison of the published results feasible.
MIT Press eBooksAdvances in Distributed and Parallel Knowledge Discovery
233 Citations2000Hillol Kargupta, Philip K. Chan
This book presents introductions to DKD and PKD, extensive reviews of the field, and state-of-the-art techniques that address the problem of extracting interesting associations, classifiers, clusters, and other patterns from data.
Neural Information Processing SystemsBoosting Decision Trees
231 Citations1995Harris Drucker, Corinna Cortes
A constructive, incremental learning system for regression problems that models data by means of locally linear experts that does not compete for data during learning and derives asymptotic results for this method.
An Alternative Model for Mixtures of Experts
216 Citations1994Lei Xu, Michael I. Jordan +1 more
An alternative model for mixtures of experts which uses a different parametric form for the gating network, trained by the EM algorithm, and which yields faster convergence.
Journal of Intelligent Information SystemsOn the Accuracy of Meta-learning for Scalable Data Mining
160 Citations1997Philip K. Chan, Salvatore J. Stolfo
This paper focussed primarily on issues related to the accuracy and efficacy of meta-learning as ageneral strategy, demonstrating that meta- learning is technically feasible in wide-area, network computing environments.
Elsevier eBooksA Comparative Evaluation of Voting and Meta-learning on Partitioned Data
148 Citations1995Philip K. Chan, Salvatore J. Stolfo
This paper evaluates different techniques for learning from partitioned data and the meta-learning approach is empirically compared with techniques in the literature that aim to combine multiple evidence to arrive at one prediction.
Neural Information Processing SystemsTraining Knowledge-Based Neural Networks to Recognize Genes in DNA Sequences
148 Citations1990Michiel Noordewier, Geoffrey G. Towell +1 more
This work describes the application of a hybrid symbolic/connectionist machine learning algorithm to the task of recognizing important genetic sequences and presents an experiment in molecular genetics that demonstrates the value of doing so.
arXiv (Cornell University)Linear and Order Statistics Combiners for Pattern Classification
137 Citations1999Kagan Tumer, Joydeep Ghosh
Neural ComputationImproving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks
100 Citations1992William G. Baxt
Two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert to overcome the loss of accuracy in either detection rate (sensitivity) or false alarm rate (specificity).
IEEE Journal of Oceanic EngineeringA neural network based hybrid system for detection, characterization, and classification of short-duration oceanic signals
97 Citations1992Joydeep Ghosh, L. Deuser +1 more
Methods for combining the outputs of several classifiers to yield a more accurate labeling are proposed and evaluated, which lead to higher classification accuracy and provide a mechanism for recognizing deviant signals and false alarms.
eScholarship (California Digital Library)On the link between error correlation and error reduction in decision tree ensembles
73 Citations1995Kamal Ali, Michael J. Pazzani
Decimated input ensembles for improved generalization
45 Citations2003Kagan Turner, Nikunj C. Oza
The need to reduce the correlation among the component classifiers is highlighted and one method for correlation reduction is investigated: input decimation, a method that uses the discriminating features of the inputs to decouple classifiers.
Pattern Recognition LettersA connectionist method for pattern classification with diverse features
42 Citations1998Ke Chen
A novel connectionist method to simultaneously use diverse features in an optimal way for pattern classification through use of soft competition among diverse features and an Expectation-Maximization (EM) learning algorithm is developed.
Combining Neural Network Regression Estimates with Regularized Linear Weights
42 Citations1996Christopher J. Merz, Michael J. Pazzani
An evaluation of the new approach, PCR*, based on principal components regression, reveals that it was the most robust combination method as the redundancy of the learned models increased, and redundancy could be handled without eliminating any of the learning models.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Evidence combination techniques for robust classification of short-duration oceanic signals</title>
15 Citations1992Joydeep Ghosh, Steven D. Beck +1 more
Four approaches to evidence combination are presented and compared using realistic oceanic data and all give better results than those obtained by the best individual classifier, and also provide a basis for detecting outliers and 'false alarms'.