EVALUATION OF CLASSIFIERS FOR AN UNEVEN CLASS DISTRIBUTION PROBLEM
Applied Artificial IntelligencePublished 30 April 2006
Sophia Daskalaki, Ioannis Kopanas, Nikolaos Avouris
Citations168
SJR quartileQ2
SJR score0.78
SNIP1.41
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This study concludes to a framework that provides the ‘best’ classifiers, identifies the performance measures that should be used as the decision criterion, and suggests the “best” class distribution based on the value of the relative gain from correct classification in the positive class.
Abstract
Classification problems with uneven class distributions present several difficulties during the training as well as during the evaluation process of classifiers. A classification problem with such characteristics has resulted from a data-mining project where the objective was to predict customer insolvency. Using the dataset from the customer insolvency problem we study several alternative methodologies which have been reported to better suit the specific characteristics of this type of problems. Three different but equally important directions are examined; (a) the performance measures that should be used for problems in this domain, (b) the class distributions that should be used for the training data sets, (c) the classification algorithms to be used. The final evaluation of the resulting classifiers is based on a study of the economic impact of classification results. This study concludes to a framework that provides the “best ” classifiers, identifies the performance measures that should be used as the decision criterion and suggests the “best ” class distribution based on the value of the relative gain from correct classification in the positive class. This framework has been applied in the customer insolvency problem, but it is claimed that it can be applied to many similar problems with uneven class distributions that almost always require a multi-objective evaluation proces. Keywords: data mining, classification, imbalanced class distributions, voting algorithms, Cost-sensitive learning 1.
Keywords
Computer ScienceDecision SciencesBusiness, Management and Accounting
Journal of Artificial Intelligence ResearchSMOTE: Synthetic Minority Over-sampling Technique
31,194 Citations2002Nitesh V. Chawla, Kevin W. Bowyer +2 more
A combination of the method of oversampling the minority (abnormal) class and under-sampling the majority class can achieve better classifier performance (in ROC space) and a combination of these methods and the area under the Receiver Operating Characteristic curve (AUC) and the ROC convex hull strategy is evaluated.
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
A study of cross-validation and bootstrap for accuracy estimation and model selection
10,733 Citations1995Ron Kohavi
The results indicate that for real-word datasets similar to the authors', the best method to use for model selection is ten fold stratified cross validation even if computation power allows using more folds.
Neural NetworksStacked generalization
7,478 Citations1992David H. Wolpert
The conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate.
International Conference on Neural Information ProcessingAdvances in kernel methods: support vector learning
5,815 Citations1999Bernhard Schölkopf, Christopher J. C. Burges +1 more
Support vector machines for dynamic reconstruction of a chaotic system, Klaus-Robert Muller et al pairwise classification and support vector machines, Ulrich Kressel.
The MIT Press eBooksFast Training of Support Vector Machines Using Sequential Minimal Optimization
5,462 Citations1998John Platt
Intelligent Data AnalysisThe class imbalance problem: A systematic study1
3,298 Citations2002Nathalie Japkowicz, Shaju Stephen
The assumption that the class imbalance problem does not only affect decision tree systems but also affects other classification systems such as Neural Networks and Support Vector Machines is investigated.
Machine LearningAn Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization
2,950 Citations2000Thomas G. Dietterich
The experiments show that in situations with little or no classification noise, randomization is competitive with (and perhaps slightly superior to) bagging but not as accurate as boosting, and sometimes better than randomization.
Machine LearningAn Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants
2,644 Citations1999Eric Bauer, Ron Kohavi
It is found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit, and that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference.
Addressing the Curse of Imbalanced Training Sets: One-Sided Selection.
2,171 Citations1997Miroslav Kubát, Stan Matwin
The foundations of cost-sensitive learning
1,835 Citations2001Charles Elkan
It is argued that changing the balance of negative and positive training examples has little effect on the classifiers produced by standard Bayesian and decision tree learning methods, and the recommended way of applying one of these methods is to learn a classifier from the training set and then to compute optimal decisions explicitly using the probability estimates given by the classifier.
Journal of the Royal Statistical Society Series C (Applied Statistics)Ridge Estimators in Logistic Regression
1,706 Citations1992Saskia le Cessie, Hans C. van Houwelingen
It is shown how ridge estimators can be used in logistic regression to improve the parameter estimates and to diminish the error made by further predictions and to predict new observations more accurately.
MetaCost
1,294 Citations1999Pedro Domingos
A principled method for making an arbitrary classifier cost-sensitive by wrapping a cost-minimizing procedure around it is proposed, called MetaCost, which treats the underlying classifier as a black box, requiring no knowledge of its functioning or change to it.
Machine LearningMachine Learning for the Detection of Oil Spills in Satellite Radar Images
1,253 Citations1998Miroslav Kubát, Robert C. Holte +1 more
This case study relates issues as problem formulation, selection of evaluation measures, and data preparation to properties of the oil spill application, such as its imbalanced class distribution, that are shown to be common to many applications.
Machine LearningRobust Classification for Imprecise Environments
1,206 Citations2001Foster Provost, Tom Fawcett
It is shown that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions, and in some cases, the performance of the hybrid actually can surpass that of the best known classifier.
A Sequential Algorithm for Training Text Classifiers
1,158 Citations1994David Lewis, William A. Gale
An algorithm for sequential sampling during machine learning of statistical classifiers was developed and tested on a newswire text categorization task and reduced by as much as 500-fold the amount of training data that would have to be manually classified to achieve a given level of effectiveness.
Lecture notes in computer scienceApplying Support Vector Machines to Imbalanced Datasets
1,132 Citations2004Rehan Akbani, Stephen Kwek +1 more
An algorithm is proposed based on a variant of the SMOTE algorithm by Chawla et al, combined with Veropoulos et al's different error costs algorithm for overcoming problems of imbalanced datasets in which negative instances heavily outnumber the positive instances.
The Case against Accuracy Estimation for Comparing Induction Algorithms
1,024 Citations1998Foster Provost, Tom Fawcett +1 more
This work describes and demonstrates what it believes to be the proper use of ROC analysis for comparative studies in machine learning research, and argues that this methodology is preferable both for making practical choices and for drawing conclusions.
Computational IntelligenceA Multiple Resampling Method for Learning from Imbalanced Data Sets
1,022 Citations2004Andrew Estabrooks, Taeho Jo +1 more
It is concluded that combining different expressions of the resampling approach is an effective solution to the tuning problem and the proposed combination scheme is evaluated on imbalanced subsets of the Reuters‐21578 text collection and is shown to be quite effective for these problems.
IEEE Transactions on Pattern Analysis and Machine IntelligenceCombination of multiple classifiers using local accuracy estimates
949 Citations1997K. Woods, W. Philip Kegelmeyer +1 more
Journal of Artificial Intelligence ResearchLearning When Training Data are Costly: The Effect of Class Distribution on Tree Induction
928 Citations2003Gary M. Weiss, Foster Provost
A "budget-sensitive" progressive sampling algorithm is introduced for selecting training examples based on the class associated with each example and it is shown that the class distribution of the resulting training set yields classifiers with good (nearly-optimal) classification performance.
Data Mining and Knowledge DiscoveryAdaptive Fraud Detection
873 Citations1997Tom Fawcett, Foster Provost
This paper uses a rule-learning program to uncover indicators of fraudulent behavior from a large database of customer transactions, which are used to create a set of monitors, which profile legitimate customer behavior and indicate anomalies.
C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling beats Over-Sampling
833 Citations2003Chris Drummond, Robert C. Holte
This paper shows that using C4.5 with undersampling establishes a reasonable standard for algorithmic comparison, and it is recommended that the cheapest class classifier be part of that standard as it can be better than under-sampling for relatively modest costs.
Lecture notes in computer scienceImproving Identification of Difficult Small Classes by Balancing Class Distribution
740 Citations2001Jorma Laurikkala
The results suggest that NCL is a useful method for improving the modeling of difficult small classes, and for building classifiers to identify these classes from the real-world data.
Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions
723 Citations1997Foster Provost, Tom Fawcett
The ROC convex hull method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers to present a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs.
IEEE Intelligent Systems and their ApplicationsDistributed data mining in credit card fraud detection
538 Citations1999Philip K. Chan, Wei Fan +2 more
The proposed methods of combining multiple learned fraud detectors under a "cost model" are general and demonstrably useful; the empirical results demonstrate that they can significantly reduce loss due to fraud through distributed data mining of fraud models.
Learning and making decisions when costs and probabilities are both unknown
455 Citations2001Bianca Zadrozny, Charles Elkan
This work explains how to obtain unbiased estimators for example-dependent costs, taking into account the difficulty that in general, probabilities and costs are not independent random variables, and the training examples for which costs are known are not representative of all examples.
Toward scalable learning with non-uniform class and cost distributions: a case study in credit card fraud detection
448 Citations1998Philip K. Chan, Salvatore J. Stolfo
A multi-classifier meta-learning approach to address very large databases with skewed class distributions and non-uniform cost per error and empirical results indicate that the approach can significantly reduce loss due to illegitimate transactions.
Exploiting the Cost (In)sensitivity of Decision Tree Splitting Criteria
202 Citations2000Chris Drummond, Robert C. Holte
Investigation of how the splitting criteria and pruning methods of decision tree learning algorithms are influenced by misclassification costs or changes to the class distribution findsSplitting criteria that are relatively insensitive to costs are found to perform as well as or better than, in terms of expected mis classification cost, splitting criteria that is cost sensitive.
Explicitly representing expected cost
186 Citations2000Chris Drummond, Robert C. Holte
There is a point/line duality between the two representations of ROC representation, allowing most techniques used in ROC analysis to be readily reproduced in the cost space.
Journal of Intelligent Information SystemsOn the Accuracy of Meta-learning for Scalable Data Mining
160 Citations1997Philip K. Chan, Salvatore J. Stolfo
This paper focussed primarily on issues related to the accuracy and efficacy of meta-learning as ageneral strategy, demonstrating that meta- learning is technically feasible in wide-area, network computing environments.
Learning goal oriented Bayesian networks for telecommunications risk management
127 Citations1996Kazuo J. Ezawa, Moninder Singh +1 more
It is argued and demonstrated that current Bayesian network learning methods may fail to perform satisfactorily in real life applications since they do not learn models tailored to a specific goal or purpose.
Applied Artificial IntelligenceREPRESENTATION DESIGN AND BRUTE-FORCE INDUCTION IN A BOEING MANUFACTURING DOMAIN
126 Citations1994Patricia Riddle, Richard Segal +1 more
This work applied inductive classification techniques to data collected in a Boeing plant with the goal of uncovering possible flaws in the manufacturing process, and formulated an algorithm that conducts an exhaustive, depth-bounded search for accurate predictive rules.
IEEE ExpertConstructing Bayesian networks to predict uncollectible telecommunications accounts
94 Citations1996Kazuo J. Ezawa, Steven W. Norton
Software that builds Bayesian network models for predicting whether a customer account or transaction is collectible and how varying model parameters and hence model structure can affect predictive accuracy are described.
The effect of class distribution on classifier learning
86 Citations2001Gary M. Weiss, Foster Provost
European Journal of Operational ResearchData mining for decision support on customer insolvency in telecommunications business
83 Citations2002Sophia Daskalaki, Ioannis Kopanas +2 more
The reported findings are very promising, making the proposed model a useful tool in the decision making process, while some of the discussed problems and limitations are of interest to researchers who intend to use data mining approaches in other similar real-life problems.
Lecture notes in computer scienceThe Role of Domain Knowledge in a Large Scale Data Mining Project
74 Citations2002Ioannis Kopanas, Nikolaos Avouris +1 more
The data mining process is described in the paper as a continuous interaction between explicit domain knowledge, and knowledge that is discovered through the use of data mining algorithms.
Diabetes Research and Clinical PracticeWhat ROC Curves Can't Do (and Cost Curves Can).
42 Citations2004Chris Drummond, Robert C. Holte
It is shown that ROC curves, as a method of visualizing classifier performance, are inadequate for the needs of Artificial Intelligence researchers in several significant respects, and it is demonstrated that a different way of visualized performance – the cost curves introduced by Drummond and Holte at KDD’2000 – overcomes these deficiencies.
Decision Support SystemsBridging the gap between business objectives and parameters of data mining algorithms
22 Citations1997Özden Gür Ali, William A. Wallace
This paper proposes and applies an experimental approach using a classification algorithm, Probabilistic Inductive Learning, to develop a mapping between the performance measures and the parameters of the algorithm used to analyze the data.
Combining Models to Improve Classifier Accuracy and Robustness 1
19 Citations1999Dean Abbott
This paper summarizes the current direction of research in combining models, and then demonstrates a process for combining models from diverse algorithm families.
Towards a theoretical framework for ensemble classification
12 Citations2003Alexander K. Seewald
This paper offers an operational reformulation of common ensemble learning schemes - Voting, Selection by Crossvalidation (X-Val), Grading and Bagging - as a Stacking scheme with appropriate parameter settings so that from a theoretical point of view all these schemes can be reduced to Stacking with an appropriate combination method.