Classifier evaluation under limited resources
Pattern Recognition LettersPublished 23 May 2006
Reuven Arbel, Lior Rokach
Citations33
SJR quartileQ1
SJR score1.00
SNIP1.43
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Performance measures that suit probabilistic classification are reviewed and two novel performance measures that can be used effectively for this task are introduced.
Abstract
Existing evaluations measures are insufficient when probabilistic classifiers are used for choosing objects to be included in a limited quota. This paper reviews performance measures that suit probabilistic classification and introduce two novel performance measures that can be used effectively for this task. It then investigates when to use each of the measures and what purpose each one of them serves. The use of these measures is demonstrated on a real life dataset obtained from the human resource field and is validated on set of benchmark datasets.
Keywords
Computer Science
Elsevier eBooksData Mining: Practical Machine Learning Tools and Techniques
25,718 Citations2011Ian H. Witten, Eibe Frank
Pattern RecognitionThe use of the area under the ROC curve in the evaluation of machine learning algorithms
7,233 Citations1997Andrew P. Bradley
AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities.
Data Mining and Knowledge Discovery Handbook
2,202 Citations2010Oded Maimon, Lior Rokach
Information Visualization in Data Mining and Knowledge DiscoveryMagnetic Bubble TechnologyStatistical Data AnalyticsUrban InformaticsAdvances in Knowledge Discovery and Data Mining data mining and knowledge discovery technologies.
Addressing the Curse of Imbalanced Training Sets: One-Sided Selection.
2,171 Citations1997Miroslav Kubát, Stan Matwin
Data Mining and Knowledge Discovery Handbook
1,535 Citations2005Oded Maimon, Lior Rokach
This book first surveys, then provides comprehensive yet concise algorithmic descriptions of methods, including classic methods plus the extensions and novel methods developed recently.
Machine LearningRobust Classification for Imprecise Environments
1,206 Citations2001Foster Provost, Tom Fawcett
It is shown that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions, and in some cases, the performance of the hybrid actually can surpass that of the best known classifier.
The Case against Accuracy Estimation for Comparing Induction Algorithms
1,024 Citations1998Foster Provost, Tom Fawcett +1 more
This work describes and demonstrates what it believes to be the proper use of ROC analysis for comparative studies in machine learning research, and argues that this methodology is preferable both for making practical choices and for drawing conclusions.
Machine LearningError Reduction through Learning Multiple Descriptions
249 Citations1994Kamal Ali, Michael J. Pazzani
It is empirically shown that it is possible to learn descriptions that make less correlated errors in domains in which many ties in the search evaluation measure (e.g. information gain) are experienced during learning.
Computational Statistics & Data AnalysisNoisy replication in skewed binary classification
92 Citations2000Sauchi Stephen Lee
The effect of class distribution on classifier learning
86 Citations2001Gary M. Weiss, Foster Provost
Magical thinking in data mining
67 Citations2001Charles Elkan
Too many researchers and practitioners in data mining do not appreciate properly the issue of statistical significance and the danger of overfitting and it is pointless to apply a very complicated learning algorithm, or to perform a very time-consuming model search.
Comparisons of classification methods for screening potential compounds
13 Citations2002Aijun An, Yuanyuan Wang
A number of data mining and statistical methods are compared on the drug design problem of modeling molecular structure-activity relationships, which can be used to identify active compounds based on their chemical structures from a large inventory of chemical compounds.
Data Mining for Target Marketing
8 Citations2009Nissan Levin, Jacob Zahavi
This chapter discusses how Data Mining modeling and analysis can support targeting applications, focusing on three types of targeting models: continuous- choice models, discrete-choice models and in-market timing models, discussing alternative modeling for each application and decision making.