Relating ensemble diversity and performance: A study in class noise detection
NeurocomputingPublished 10 February 2015
Borut Sluban, Nada Lavrač
Citations65
SJR quartileQ1
SJR score1.47
SNIP1.94
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper explores the relation between ensemble diversity and noise detection performance in the context of ensemble-based class noise detection by studying different diversity measures on a range of heterogeneous noise detection ensembles.
Abstract
The advantage of ensemble methods over single methods is their ability to correct the errors of individual ensemble members and thereby improve the overall ensemble performance. This paper explores the relation between ensemble diversity and noise detection performance in the context of ensemble-based class noise detection by studying different diversity measures on a range of heterogeneous noise detection ensembles. In the empirical analysis the majority and the consensus ensemble voting schemes are studied. It is shown that increased diversity of ensembles using the majority voting scheme does not lead to better noise detection performance and may even degrade the performance of heterogeneous noise detection ensembles. On the other hand, for consensus-based noise detection ensembles the results show that more diverse ensembles achieve higher precision of class noise detection, whereas less diverse ensembles lead to higher recall of noise detection and higher F-scores.
Keywords
Computer ScienceDecision SciencesEngineering
Machine LearningRandom Forests
126,192 Citations2001Leo Breiman
Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Wiley series in probability and statisticsStatistical Methods for Rates and Proportions
18,048 Citations2003Joseph L. Fleiss, Bruce Levin +1 more
ACM SIGKDD Explorations NewsletterThe WEKA data mining software
17,849 Citations2009Mark Hall, Eibe Frank +4 more
This paper provides an introduction to the WEKA workbench, reviews the history of the project, and, in light of the recent 3.6 stable release, briefly discusses what has been added since the last stable version (Weka 3.4) released in 2003.
Machine LearningBagging predictors
16,377 Citations1996Leo Breiman
Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy.
Statistical Comparisons of Classifiers over Multiple Data Sets
11,198 Citations2006Janez Demšar
A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
Lecture notes in computer scienceEnsemble Methods in Machine Learning
7,862 Citations2000Thomas G. Dietterich
Neural NetworksStacked generalization
7,478 Citations1992David H. Wolpert
The conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate.
Isolation Forest
5,601 Citations2008Fei Tony Liu, Kai Ming Ting +1 more
The use of isolation enables the proposed method, iForest, to exploit sub-sampling to an extent that is not feasible in existing methods, creating an algorithm which has a linear time complexity with a low constant and a low memory requirement.
Machine LearningThe Strength of Weak Learnability
3,293 Citations1990Robert E. Schapire
In this paper, a method is described for converting a weak learning algorithm into one that achieves arbitrarily high accuracy, and it is shown that these two notions of learnability are equivalent.
Proceedings of the Royal Society of LondonVII. Note on regression and inheritance in the case of two parents
3,237 Citations1895Karl Pearson
Consider a population in which sexual selection and natural selection may or may not be taking place, and the following expressions may be shown to give the law of inheritance of the population.
TechnometricsCombining Pattern Classifiers: Methods and Algorithms
3,234 Citations2005Subhash Bagui
This chapter discusses the development of the Spatial Point Pattern Analysis Code in S–PLUS, which was developed in 1993 by P. J. Diggle and D. C. Griffith.
Machine LearningThe strength of weak learnability
2,434 Citations1990Robert E. Schapire
In this paper, it is shown that the two notions of learnability are equivalent, and a method is described for converting a weak learning algorithm into one that achieves arbitrarily high accuracy.
Machine LearningMeasures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
2,288 Citations2003Ludmila I. Kuncheva, Christopher J. Whitaker
Although there are proven connections between diversity and accuracy in some special cases, the results raise some doubts about the usefulness of diversity measures in building classifier ensembles in real-life pattern recognition problems.
ACM Transactions on Knowledge Discovery from DataIsolation-Based Anomaly Detection
1,997 Citations2012Fei Tony Liu, Kai Ming Ting +1 more
This article proposes a method called Isolation Forest (iForest), which detects anomalies purely based on the concept of isolation without employing any distance or density measure---fundamentally different from all existing methods.
IEEE Transactions on Neural Networks and Learning SystemsClassification in the Presence of Label Noise: A Survey
1,721 Citations2014Benoît Frénay, Michel Verleysen
In this paper, label noise consists of mislabeled instances: no additional information is assumed to be available like e.g., confidences on labels.
Journal of Machine Learning ResearchOrange: data mining toolbox in python
1,637 Citations2013Janez Demšar, Tomaž Curk +14 more
Orange is a machine learning and data mining suite for data analysis through Python scripting and visual programming, which features interactive data analysis and component-based assembly of data mining procedures.
Information FusionDiversity creation methods: a survey and categorisation
954 Citations2004Gavin Brown, Jeremy Wyatt +2 more
This paper reviews the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature, and introduces the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied.
Artificial Intelligence ReviewClass Noise vs. Attribute Noise: A Quantitative Study
896 Citations2004Xingquan Zhu, Xindong Wu
A systematic evaluation on the effect of noise in machine learning separates noise into two categories: class noise and attribute noise, and investigates the relationship between attribute noise and classification accuracy, the impact of noise at different attributes, and possible solutions in handling attribute noise.
Ensemble selection from libraries of models
844 Citations2004Rich Caruana, Alexandru Niculescu-Mizil +2 more
A method for constructing ensembles from libraries of thousands of models using forward stepwise selection to be optimized to performance metric such as accuracy, cross entropy, mean precision, or ROC Area is presented.
Bias plus variance decomposition for zero-one loss functions
627 Citations1996Ron Kohavi, David H. Wolpert
It is shown that in practice the naive frequency based estimation of the decompo sition terms is by itself biased and how to correct for this bias is correct.
Journal of Artificial Intelligence ResearchIdentifying Mislabeled Training Data
600 Citations1999C. E. Brodley, M. A. Friedl
This paper uses a set of learning algorithms to create classifiers that serve as noise filters for the training data and suggests that for situations in which there is a paucity of data, consensus filters are preferred, whereas majority vote filters are preferable for situations with an abundance of data.
Diversity analysis on imbalanced data sets by using ensemble models
546 Citations2009Shuo Wang, Xin Yao
This paper explores the impact of diversity on each class and overall performance and improves SMOTE in a novel way for solving multi-class data sets in ensemble model - SMOTEBagging.
ACM Transactions on Information SystemsA critical investigation of recall and precision as measures of retrieval system performance
471 Citations1989Vijay V. Raghavan, P. Bollmann +1 more
This paper systematically investigates the various problems and issues associated with the use of recall and precision as measures of retrieval system performance and provides a comparative analysis of methods available for defining precision in a probabilistic sense to promote a better understanding of the various issues involved in retrieval performance evaluation.
Machine LearningAn analysis of diversity measures
407 Citations2006E. K. Tang, Ponnuthurai Nagaratnam Suganthan +1 more
A theoretical analysis on six existing diversity measures is presented, show underlying relationships between them, and relate them to the concept of margin, which is more explicitly related to the success of ensemble learning algorithms.
Philosophical Transactions of the Royal Society of London Series A Containing Papers of a Mathematical or Physical CharacterVII. On the association of attributes in statistics: with illustrations from the material of the childhood society, &c
400 Citations1900George Yule
In the ordinary theory of statistical correlation, normal or otherwise, material susceptible of continuous variation, or at least of variation by a considerable number of discontinuous steps, certain practical cases arise, where either no variation is thinkable at all, or else is not measured or possibly measurable.
Pattern Analysis and ApplicationsLimits on the majority vote accuracy in classifier fusion
352 Citations2003Ludmila I. Kuncheva, Christopher J. Whitaker +2 more
It is shown that the majority vote with dependent classifiers can potentially offer a dramatic improvement both over independent classifiers and over an individual classifier with accuracy p, and a functional relationship between the limits and the pairwise dependence Q is derived.
IEEE Transactions on Pattern Analysis and Machine IntelligenceA theoretical and experimental analysis of linear combiners for multiple classifier systems
309 Citations2005Giorgio Fumera, Fabio Roli
This analysis focuses on the simplest and most widely used implementation of linear combiners, which consists of assigning a nonnegative weight to each individual classifier, and considers the ideal performance of this combining rule, i.e., that achievable when the optimal values of the weights are used.
Machine LearningUsing Correspondence Analysis to Combine Classifiers
210 Citations1999Christopher J. Merz
The method described uses the strategies of stacking and Correspondence Analysis to model the relationship between the learning examples and their classification by a collection of learned models, and a nearest neighbor method is applied within the resulting representation to classify previously unseen examples.
Lecture notes in computer scienceUsing Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error
182 Citations2001Gabriele Zenobi, Pádraig Cunningham
This paper presents a process for producing ensembles of classifiers based on different feature subsets that emphasizes diversity (ambiguity) in the ensemble members and finds that the ensemble based on ambiguity have lower generalization error.
Experiments with Noise Filtering in a Medical Domain
176 Citations1999Dragan Gamberger, Nada Lavrač +1 more
Lecture notes in computer scienceEnsemble Methods for Noise Elimination in Classification Problems
170 Citations2003Sofie Verbaeten, Anneleen Van Assche
This paper considers the problem of mislabeled training examples in classification tasks, and addresses this problem by pre-processing the training set, i.e. by identifying and removing outliers from theTraining set.
Decision Tree Ensemble: Small Heterogeneous Is Better Than Large Homogeneous
154 Citations2008Mike Gashler, Christophe Giraud-Carrier +1 more
The results indicate that random forests do poorly when faced with irrelevant attributes, while the heterogeneous technique handles them robustly, and it is shown that large ensembles of random trees are more susceptible to diminishing returns than the technique.
ProceedingsGetting the Most Out of Ensemble Selection
143 Citations2006Rich Caruana, Art Munson +1 more
Improved ensemble selection methods are presented that double the benefit of the original method, and the performance impact of pruning the number of models available for ensemble selection is studied.
Lecture notes in computer scienceDiversity Regularized Ensemble Pruning
139 Citations2012Nan Li, Yang Yu +1 more
A theoretical study on the effect of diversity on the generalization performance of voting in the PAC-learning framework and applies explicit diversity regularization to ensemble pruning, and proposes the Diversity Regularized Ensemble Pruning (DREP) method.
Lecture notes in computer science“Good” and “Bad” Diversity in Majority Vote Ensembles
118 Citations2010Gavin Brown, Ludmila I. Kuncheva
A decomposition of the majority vote error into three terms: average individual accuracy, “good” diversity and “bad diversity”, which is taken out of the individual error whereas the bad diversity term is added to it.
Artificial Intelligence in MedicineActive subgroup mining: a case study in coronary heart disease risk group detection
86 Citations2003Dragan Gamberger, Nada Lavrač +1 more
The approach proposes active expert involvement in the following steps of the knowledge discovery process: data gathering, cleaning and transformation, subgroup discovery, statistical characterization of induced subgroups, their interpretation, and the evaluation of results.
Data Mining and Knowledge DiscoveryEnsemble-based noise detection: noise ranking and visual performance evaluation
83 Citations2013Borut Sluban, Dragan Gamberger +1 more
An ensemble-based noise ranking methodology for explicit noise and outlier identification, named Noise-Rank, which was successfully applied to a real-life medical problem as proven in domain expert evaluation and a methodology for visual performance evaluation of noise detection algorithms in the precision-recall space, named Viper are presented.
NeurocomputingPruning an ensemble of classifiers via reinforcement learning
81 Citations2008Ioannis Partalas, Grigorios Tsoumakas +1 more
A new pruning approach that uses the Q-learning algorithm in order to approximate an optimal policy of choosing whether to include or exclude each classifier from the ensemble is contributed.
Data Mining and Knowledge DiscoveryFRaC: a feature-modeling approach for semi-supervised and unsupervised anomaly detection
78 Citations2011Keith Noto, Carla E. Brodley +1 more
It is shown that FRaC is a superior approach for the unsupervised as well as the semi-supervised anomaly detection task, compared to well-known state-of-the-art anomaly detection methods, LOF and one-class support vector machines, and to an existing feature-modeling approach.
Knowledge and Information SystemsThe pairwise attribute noise detection algorithm
76 Citations2006Jason D. Van Hulse, Taghi M. Khoshgoftaar +1 more
This work presents a novel approach for detecting instances with attribute noise and demonstrates its usefulness with case studies using two different real-world software measurement data sets, showing that PANDA provides better noise detection performance than the DM algorithm.
Intelligent Data AnalysisEnhancing software quality estimation using ensemble-classifier based noise filtering
76 Citations2005Taghi M. Khoshgoftaar, Shi Zhong +1 more
The idea is to eliminate instances that are likely to be noisy, and train classification models on "clean" data, and preliminary results suggest that the 25 classifiers may be effectively reduced to 13.
Frontiers in artificial intelligence and applicationsFocused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection
71 Citations2008Ioannis Partalas, Grigorios Tsoumakas +1 more
Experimental comparison of the proposed method, dubbed Focused Ensemble Selection (FES), against state-of-the-art greedy ensemble selection methods shows that it leads to small ensembles with high predictive performance.
Anomaly Detection Using an Ensemble of Feature Models
40 Citations2010Keith Noto, Carly Brodley +1 more
This work has developed a novel, information-theoretic anomaly measure that selects against noisy and irrelevant features and significantly improves performance over current state-of-the-art feature space distance and density-based approaches.
Frontiers in artificial intelligence and applicationsAdvances in Class Noise Detection
38 Citations2010Borut Sluban, Dragan Gamberger +1 more
This work aims at detecting noisy instances to be inspected by the domain expert in the phase of data understanding, and should have high precision of class noise detection, where the precision-recall trade-off is modeled using the F-measure.
A boosting approach to remove class label noise
38 Citations2005Amitava Karmaker, Stephen Kwek
International Journal of Hybrid Intelligent SystemsA boosting approach to remove class label noise1
18 Citations2006Amitava Karmaker, Stephen Kwek
A modification to AdaBoost is proposed that is more tolerant to class label noise, which further enhances its ability to boost prediction accuracy.
Lecture notes in computer scienceDiversity in Combinations of Heterogeneous Classifiers
18 Citations2009Kuo-Wei Hsu, Jaideep Srivastava
The use of combinations of heterogeneous classifiers to achieve better diversity is introduced and the relationship between heterogeneity and diversity is studied.
Drift detection and monitoring in non-stationary environments
15 Citations2014Imen Khamassi, M. Sayed-Mouchaweh
In this paper a new error distance based approach for drift detection and monitoring, namely EDIST, is proposed, and a difference in error distance distributions of two data-windows is monitored through a statistical hypothesis test.
International Journal of Software Engineering and Knowledge EngineeringDETECTING NOISY INSTANCES WITH THE ENSEMBLE FILTER: A STUDY IN SOFTWARE QUALITY ESTIMATION
14 Citations2006Taghi M. Khoshgoftaar, Vedang H. Joshi +1 more
A noise handling technique that attempts to improve the quality of datasets for classification purposes by eliminating instances that are likely to be noise by using twenty five different classification techniques to create an ensemble filter for eliminating likely noise.
Annual Conference of the PHM SocietyEnsemble classifiers for drift detection and monitoring in dynamical environments
9 Citations2013Imen Khamassi, Moamar Sayed‐Mouchaweh +2 more
Two ensembles learning algorithms are presented, which respectively combine the Online Bagging and the Online Boosting with the drift detection method EDIST, to develop an ensemble learning algorithm which explicitly handles concept drifts by providing useful descriptions about location, speed and severity of drifts.
Using good and bad diversity measures in the design of ensemble systems: A genetic algorithm approach
4 Citations2013Antonino A. Feitosa Neto, Anne M. P. Canuto +1 more
This paper investigates the influence of measures of good and bad diversity when used explicitly to guide the search of a genetic algorithm to design ensemble systems, and aims to determine which set of objectives generates more accurate ensembles.