An introduction to ROC analysis
Pattern Recognition LettersPublished 22 December 2005
Tom Fawcett
Citations21,254
SJR quartileQ1
SJR score1.00
SNIP1.43
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.
Abstract
Receiver operating characteristics (ROC) graphs are useful for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been used increasingly in machine learning and data mining research. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.
Keywords
Computer ScienceHealth Professions
BiometricsClassification and Regression Trees.
23,841 Citations1984Alexander Gordon, Leo Breiman +3 more
RadiologyThe meaning and use of the area under a receiver operating characteristic (ROC) curve.
21,812 Citations1982J A Hanley, Barbara J. McNeil
A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics, is presented and it is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a random chosen non-diseased subject.
ScienceMeasuring the Accuracy of Diagnostic Systems
10,020 Citations1988John A. Swets
For diagnostic systems used to distinguish between two classes of events, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy.
Pattern RecognitionThe use of the area under the ROC curve in the evaluation of machine learning algorithms
7,233 Citations1997Andrew P. Bradley
AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities.
Machine LearningA Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
2,264 Citations2001David J. Hand, Robert John Till
This work extends the definition of the area under the ROC curve to the case of more than two classes by averaging pairwise comparisons and proposes an alternative definition of proportion correct based on pairwise comparison of classes for a simple artificial case.
MetaCost
1,294 Citations1999Pedro Domingos
A principled method for making an arbitrary classifier cost-sensitive by wrapping a cost-minimizing procedure around it is proposed, called MetaCost, which treats the underlying classifier as a black box, requiring no knowledge of its functioning or change to it.
Machine LearningMachine Learning for the Detection of Oil Spills in Satellite Radar Images
1,253 Citations1998Miroslav Kubát, Robert C. Holte +1 more
This case study relates issues as problem formulation, selection of evaluation measures, and data preparation to properties of the oil spill application, such as its imbalanced class distribution, that are shown to be common to many applications.
Machine LearningRobust Classification for Imprecise Environments
1,206 Citations2001Foster Provost, Tom Fawcett
It is shown that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions, and in some cases, the performance of the hybrid actually can surpass that of the best known classifier.
The Case against Accuracy Estimation for Comparing Induction Algorithms
1,024 Citations1998Foster Provost, Tom Fawcett +1 more
This work describes and demonstrates what it believes to be the proper use of ROC analysis for comparative studies in machine learning research, and argues that this methodology is preferable both for making practical choices and for drawing conclusions.
Data Mining and Knowledge DiscoveryAdaptive Fraud Detection
873 Citations1997Tom Fawcett, Foster Provost
This paper uses a rule-learning program to uncover indicators of fraudulent behavior from a large database of customer transactions, which are used to create a set of monitors, which profile legitimate customer behavior and indicate anomalies.
Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions
723 Citations1997Foster Provost, Tom Fawcett
The ROC convex hull method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers to present a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs.
Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers
656 Citations2001Bianca Zadrozny, Charles Elkan
It is concluded that binning succeeds in significantly improving naive Bayesian probability estimates, while for improving decision tree probability estimates the recommend smoothing by -estimation and a new variant of pruning that is called curtailment.
Elsevier eBooksSIGNAL DETECTION THEORY: VALUABLE TOOLS FOR EVALUATING INDUCTIVE LEARNING
336 Citations1989Kent A. Spackman
The use of signal detection theory as a tool for evaluating and comparing concept descriptions learned by inductive inference and the use of ROC curves are described.
Evaluating text categorization
234 Citations1991David Lewis
Combining data mining and machine learning for effective user profiling
174 Citations1996Tom Fawcett, Foster Provost
This paper combines data mining and constructive induction with more standard machine learning techniques to design methods for detecting fraudulent usage of cellular telephones based on profiling customer behavior, and uses a rule-learning program to uncover indicators of fraudulent behavior from a large database of cellular calls.
Using rule sets to maximize ROC performance
141 Citations2002Tom Fawcett
This paper empirically investigates different strategies for evaluating rule sets when the goal is to maximize the scoring (ROC) performance.
Robust classification systems for imprecise environments
121 Citations1998Foster Provost, Tom Fawcett
It is shown that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization.
Kent Academic Repository (University of Kent)Repairing concavities in ROC curves
86 Citations2005Peter Flach, Shaomin Wu
This paper investigates methods to detect and repair concavities in ROC curves by manipulating model predictions by building a hybrid model combining the two better models with an inversion of the poorer models.
Confidence Bands for ROC Curves: Methods and an Empirical Study
71 Citations2004Sofus A. Macskassy, Foster Provost
It is shown how some of these methods for generating and evaluating condence bands on ROC curves work remarkably well, others are too loose, and that existing machine learning methods for generation of 1-dimensional condence intervals do not translate well to generation of simultanous bands|their bands are too tight.
Machine LearningLearning in the “Real World”
56 Citations1998Lorenza Saitta, Filippo Neri
This paper defines and characterize the process of developing a “real-world” Machine Learning application, with its difficulties and relevant issues, distinguishing it from the popular practice of exploiting ready-to-use data sets.
Confidence Bands for ROC Curves
39 Citations2003Sofus A. Macskassy, Foster Provost +1 more
It is shown empirically that the optimized confidence bands fit much better and that, using a new evaluation method, it is possible to gauge therelative fit of different confidence bands.
Representation quality in text classification
31 Citations1990David Lewis
The results of an experiment on improving text representation quality are presented, as well as an analysis of the results and the directions they suggest for future research.
Computer Physics CommunicationsA rule-learning program in high energy physics event classification
28 Citations1991Scott H. Clearwater, Einat Stern
A rule-learning program is applied to the problem of event classification in high energy physics that can easily explore multi-dimensional rules as well as rules that may seem non-intuitive at first to the physicist.
A Method for Discovering the Insignificance of One's Best Classifier and the Unlearnability of a Classification Task
8 Citations2002George Forman
This paper presents a method to address the question of whether the chosen best classifier is significant on a held-out test set by comparing the distribution of performance scores obtained by many simple classifiers that are randomly generated.