ROC curves for regression
Pattern RecognitionPublished 25 June 2013Open access
José Hernández‐Orallo
Citations141
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new representation of regression models in the so-called regression ROC (RROC) space is presented, with the notions of optimal operating condition, convexity, dominance, and several evaluation metrics that can be shown graphically, such as the area over the RROC curve (AOC).
Abstract
“NOTICE: this is the author’s version of a work that was accepted for publication in Pattern Recognition. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Pattern Recognition Volume 46, Issue 12, December 2013, Pages 3395–3411 \nDOI: 10.1016/j.patcog.2013.06.014
Keywords
Computer ScienceHealth Professions
Pattern Recognition LettersAn introduction to ROC analysis
21,254 Citations2005Tom Fawcett
The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.
ScienceMeasuring the Accuracy of Diagnostic Systems
10,020 Citations1988John A. Swets
For diagnostic systems used to distinguish between two classes of events, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy.
Pattern RecognitionThe use of the area under the ROC curve in the evaluation of machine learning algorithms
7,233 Citations1997Andrew P. Bradley
AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities.
The foundations of cost-sensitive learning
1,835 Citations2001Charles Elkan
It is argued that changing the balance of negative and positive training examples has little effect on the classifiers produced by standard Bayesian and decision tree learning methods, and the recommended way of applying one of these methods is to learn a classifier from the training set and then to compute optimal decisions explicitly using the probability estimates given by the classifier.
Machine LearningMeasuring classifier performance: a coherent alternative to the area under the ROC curve
969 Citations2009David J. Hand
A simple valid alternative to the AUC is proposed, and the property of it being fundamentally incoherent in terms of misclassification costs is explored in detail.
Pattern Recognition LettersAn experimental comparison of performance measures for classification
888 Citations2008Cèsar Ferri, José Hernández‐Orallo +1 more
This work analyzes experimentally the behaviour of 18 different performance metrics in several scenarios, identifying clusters and relationships between measures and makes a comprehensive analysis of the relationships between metrics, and a taxonomy and arrangement of them according to the previous traits.
Journal of the American Statistical AssociationBayesian Estimation and Prediction Using Asymmetric Loss Functions
854 Citations1986Arnold Zellner
Journal of Biomedical InformaticsThe use of receiver operating characteristic curves in biomedical informatics
846 Citations2005Thomas A. Lasko, Jui G. Bhagwat +2 more
The basic concepts of ROC analysis are reviewed, their use with sample calculations, make recommendations drawn from the literature, and list readily available software are made.
Machine Learning: The Art and Science of Algorithms that Make Sense of Data
697 Citations2012Peter Flach
Peter Flach's clear, example-based approach begins by discussing how a spam filter works, which gives an immediate introduction to machine learning in action, with a minimum of technical fuss.
ROC Curves for Continuous Data
659 Citations2009W. J. Krzanowski, David J. Hand
This book provides illustrative examples of the major methodological developments and includes as much of the mathematical theory as necessary without making the treatment too dense.
Econometric TheoryOptimal Prediction Under Asymmetric Loss
346 Citations1997Peter Christoffersen, Francis X. Diebold
A key theme is that the conditionally optimal forecast is biased under asymmetric loss and that theconditionally optimal amount of bias is time varying in general and depends on higher order conditional moments.
arXiv (Cornell University)Types of Cost in Inductive Concept Learning
330 Citations2002Peter D. Turney
This paper attempts to create a taxonomy of the different types of cost that are involved in inductive concept learning, which may help to organize the literature on cost-sensitive learning.
Machine LearningCost curves: An improved method for visualizing classifier performance
324 Citations2006Chris Drummond, Robert C. Holte
This paper introduces cost curves, a graphical technique for visualizing the performance (error rate or expected cost) of 2-class classifiers over the full range of possible class distributions and misclassification costs.
Journal of the Operational Research SocietyPrediction with a Generalized Cost of Error Function
318 Citations1969Clive W. J. Granger
Learning Decision Trees Using the Area Under the ROC Curve
266 Citations2002Cèsar Ferri, Peter Flach +1 more
This paper shows how a single decision tree can represent a set of classifiers by choosing different labellings of its leaves, or equivalently, an ordering on the leaves, and proposes a novel splitting criterion which chooses the split with the highest local AUC.
The geometry of ROC space: understanding machine learning metrics through ROC isometrics
244 Citations2003Peter Flach
The paper demonstrates that the graphical depiction of machineLearning metrics by means of ROC isometrics gives many useful insights into the characteristics of these metrics, and provides a foundation on which a theory of machine learning metrics can be built.
Journal of Statistical Planning and InferenceBayesian approach to life testing and reliability estimation using asymmetric loss function
229 Citations1991Asit P. Basu, Nader Ebrahimi
Journal of the American Statistical AssociationBayesian Estimation and Prediction Using Asymmetric Loss Functions
205 Citations1986Arnold Zellner
Regression error characteristic curves
201 Citations2003Jinbo Bi, Kristin P. Bennett
The shape of the curve reveals additional information that can be used to guide modeling and users can quickly assess the relative merits of many regression functions by examining the relative position of their REC curves.
Journal of Systems and SoftwareAn empirical study of maintenance and development estimation accuracy
197 Citations2002Barbara Kitchenham, Shari Lawrence Pfleeger +2 more
Analysis of data from 145 maintenance and development projects managed by a single outsourcing company found that effort estimates chosen as a basis for project budgets were reasonably good, and there was little evidence that the accuracy of the selected estimates was due to their becoming the target values for the project managers.
Bristol Research (University of Bristol)A unified view of performance metrics: translating threshold choice into expected classification loss
191 Citations2012José Hernández‐Orallo, Peter Flach +1 more
A Coherent Interpretation of AUC as a Measure of Aggregated Classification Performance
189 Citations2011C sar Ferri, Jos Hern ndez-orallo +1 more
Should one wish to consider only optimal thresholds, it is demonstrated that a simple and more intuitive alternative to Hand's H measure is already available in the form of the area under the cost curve, both uniform and hence model-independent.
Journal of Applied EconometricsFurther results on forecasting and model selection under asymmetric loss
174 Citations1996Peter Christoffersen, Francis X. Diebold
A new technique for solving prediction problems under asymmetric loss using piecewise-linear approximations to the loss function is proposed, and the existence and uniqueness of the optimal predictor are established.
Journal of the American Statistical AssociationTesting Forecast Optimality Under Unknown Loss
173 Citations2007Andrew J. Patton, Allan Timmermann
Lecture notes in computer scienceVolume under the ROC Surface for Multi-class Problems
163 Citations2003Cèsar Ferri, José Hernández‐Orallo +1 more
This paper presents the real extension to the Area Under the AUC Curve in the form of the Volume Under the ROC Surface (VUS), showing how to compute the polytope that corresponds to the absence of classifiers, to the best classifier and to whatever set of classifier.
Spanish Economic ReviewOutline of forecast theory using generalized cost functions
153 Citations1999Clive W. J. Granger
Encyclopedia of Software EngineeringResource Estimation in Software Engineering
146 Citations2002Lionel Briand, Isabella Wieczorek
This paper presents a comprehensive overview of the state of the art in software resource estimation and describes common estimation methods and provides an evaluation framework to systematically compare and assess alternative estimation methods.
Statistics in MedicineEvaluating diagnostic tests: The area under the ROC curve and the balance of errors
139 Citations2010David J. Hand
The AUC has a well‐understood weakness when comparing ROC curves which cross, but it also has the more fundamental weakness of failing to balance different kinds of misdiagnoses effectively.
Pattern RecognitionIterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs
119 Citations2010Wael Khreich, Éric Granger +2 more
The results of computer simulations indicate that the iterative Boolean combination (IBC) of responses from multiple HMMs can achieve a significantly higher level of performance than the Boolean conjunction and disjunction combinations, especially when training data are limited and imbalanced.
Pattern RecognitionSelecting features in microarray classification using ROC curves
99 Citations2006Hiroshi Mamitsuka
A new method based on the ROC (Receiver Operating Characteristic) curve to efficiently select a feature subset in classifying a high-dimensional microarray dataset with a limited number of observations is presented.
Lecture notes in computer scienceUtility-Based Regression
92 Citations2007Luı́s Torgo, Rita P. Ribeiro
This paper describes two metrics for asserting the costs and benefits of the predictions of any model given a set of test cases, and illustrates the use of these metrics in the context of a specific type of applications where non-uniform costs are required: the prediction of rare extreme values of a continuous target variable.
Lecture notes in computer sciencePrecision and Recall for Regression
81 Citations2009Luı́s Torgo, Rita P. Ribeiro
A generalization for regression of the concepts of precision and recall often used in classification is described, which is able to focus the evaluation of predictive models on the cases that really matter for these applications.
Lecture notes in computer scienceImproving the AUC of Probabilistic Estimation Trees
79 Citations2003Cèsar Ferri, Peter Flach +1 more
A new probability smoothing that takes into account the class distributions of all the nodes from the root to each leaf is derived and some new splitting criteria aimed at improving probability estimates rather than improving classification accuracy are introduced.
Pattern RecognitionMaximizing the area under the ROC curve by pairwise feature combination
77 Citations2007Claudio Marrocco, Robert P. W. Duin +1 more
This paper proposes a nonparametric linear classifier based on the maximization of AUC that lies on the analysis of the Wilcoxon-Mann-Whitney statistic of each single feature and an iterative pairwise coupling of the features for the optimization of the ranking of the combined feature.
Brier Curves: a New Cost-Based Visualisation of Classifier Performance
61 Citations2011C sar Ferri, Jos Hern ndez-orallo +1 more
A more natural way of visualising classifier performance in cost space is proposed, which is to plot probabilistic loss on the y-axis, i.e., the loss arising from the probability estimates.
Pattern RecognitionMaximizing area under ROC curve for biometric scores fusion
55 Citations2008Kar‐Ann Toh, Jaihie Kim +1 more
This work proposes to optimize the ROC performance directly according to the fusion classifier design and proposes a smooth approximate formulation that enables a direct optimization of the AUC with respect to the classifier parameters.
IGI Global eBooksCalibration of Machine Learning Models
54 Citations2010Antonio Bella, Cèsar Ferri +2 more
Optimizing abstaining classifiers using ROC analysis
53 Citations2005Tadeusz Pietraszek
This work proposes a method to optimally build a specific type of abstaining binary classifiers using ROC analysis based on optimization criteria in the following three models: cost-based, bounded-abstention and bounded-improvement.
Decision Support SystemsAn extended tuning method for cost-sensitive regression and forecasting
48 Citations2011Huimin Zhao, Atish P. Sinha +1 more
Empirical evaluation in the loan charge-off forecasting domain demonstrates that the proposed extended tuning method for cost-sensitive regression can further lower the misprediction cost significantly.
Pattern RecognitionAdaptive ROC-based ensembles of HMMs applied to anomaly detection
46 Citations2011Wael Khreich, Éric Granger +2 more
A system based on the receiver operating characteristic (ROC) is proposed to efficiently adapt ensembles of HMMs (EoHMMs) in response to new data, according to a learn-and-combine approach, and can be employed for a wide range of classifiers and detection applications.
Journal of Management Information SystemsTuning Data Mining Methods for Cost-Sensitive Regression: A Study in Loan Charge-Off Forecasting
45 Citations2008Gaurav Bansal, Atish P. Sinha +1 more
This paper advocates the use of average misprediction cost as a measure for assessing the performance of a cost-sensitive regression model, and proposes an efficient algorithm for tuning a regression model to further reduce its average misPrediction cost.
Machine LearningROC curves in cost space
44 Citations2013José Hernández‐Orallo, Peter Flach +1 more
It is shown that ROC curves can be transferred to cost space by means of a very natural threshold choice method, which sets the decision threshold such that the proportion of positive predictions equals the operating condition and the expected loss as measured by the area under these curves is linearly related to AUC.
Knowledge and Information SystemsRanking-based evaluation of regression models
42 Citations2006Saharon Rosset, Claudia Perlich +1 more
Decision Support for Data Mining
39 Citations2003Peter Flach, Hendrik Blockeel +3 more
This chapter gives an introduction to ROC (‘receiver operating characteristics’) analysis and its applications to data mining and shows how ROC analysis can aid in constructing and refining models in the modeling stage.
Annals of the Institute of Statistical MathematicsReal estate price prediction under asymmetric loss
38 Citations1995Michael Cain, Christian Janßen
Journal of ForecastingAre Analysts' Loss Functions Asymmetric?
37 Citations2011Mark Clatworthy, David Peel +1 more
Modifying ROC Curves to Incorporate Predicted Probabilities
36 Citations2005Cèsar Ferri, Peter Flach +1 more
A new probabilistic version of AUC is introduced, called pAUC, which evaluates ranking performance, but also takes the magnitude of the probabilities into account.
Regression error characteristic surfaces
31 Citations2005Luı́s Torgo
The notion of REC surfaces are presented, how to use them to compare the performance of models, and their use with an important practical class of applications: the prediction of rare extreme values are illustrated.
Training artificial neural networks for time series prediction using asymmetric cost functions
30 Citations2002Sven F. Crone
A set of asymmetric cost functions is proposed as novel error functions for neural network training, evaluating the performance in competition to basic forecast methods using various error measures.
Machine Learning Methods and Asymmetric Cost Function to Estimate Execution Effort of Software Testing
28 Citations2010Daniel G. Silva, Mário Jino +1 more
The main contributions of this paper are the approach of selecting the significant variables for database synthesis and the use of an artificial neural network trained with an asymmetric cost function.
Pattern RecognitionAn online AUC formulation for binary classification
22 Citations2011Youngsung Kim, Kar‐Ann Toh +3 more
This paper proposes an online learning algorithm which involves a pairwise cost function which pairs up a newly arrived data point with those of opposite class in stored data and shows promising potential in terms of classification AUC, accuracy, and computational efficiency.
Pattern RecognitionThe ROC manifold for classification systems
21 Citations2010Christine M. Schubert, Steven N. Thorsen +1 more
A mathematical definition based on vector space methods to describe the ROC manifold and CC manifold as duals in a given sense, which can be used to determine the usefulness of a classification system based on a given performance criterion.
Pattern RecognitionPartial AUC maximization in a linear combination of dichotomizers
21 Citations2011Maria Teresa Ricamato, Francesco Tortorella
This paper presents a new method that aims at maximizing the pAUC by means of linear combination of classifiers, which has been proved on two biometric databases.
Pattern RecognitionROC curve estimation and hypothesis testing: applications to breast cancer detection
19 Citations1982James E. Goin
An introduction to statistical estimation and hypothesis testing methodology, which can be employed in analyzing the performance of various classifiers in a breast cancer detection task is presented.
Ranking-Based Evaluation of Regression Models
17 Citations2006Saharon Rosset, Claudia Perlich +1 more
ACM SIGKDD Explorations NewsletterThe 1st workshop on ROC analysis in artificial intelligence (ROCAI-2004)
16 Citations2004José Hernández‐Orallo, Cèsar Ferri +2 more
This short report includes a summary of the presentations and discussions held during the ROCAI-2004 workshop, as well as the workshop conclusions and the future agenda.
Boolean Combination of Classifiers in the ROC Space
13 Citations2010Wael Khreich, Éric Granger +2 more
The results of computer simulations conducted on synthetic and real-world host-based intrusion detection data indicate that combining the responses from multiple HMMs with IBC can achieve a significantly higher level of performance than with the AND and OR combinations.
Computational Statistics & Data AnalysisAn extension of the Gauss–Newton algorithm for estimation under asymmetric loss
6 Citations2004Matei Demetrescu
The provided extension of the Gauss-Newton algorithm is tailored for the associated optimization problem and can be viewed as a succession of linear generalized least-squares regressions and is easy to implement.
Journal of Machine Learning ResearchA unified view of performance metrics
6 Citations2012Hernández-OralloJosé, FlachPeter +1 more
This analysis provides a comprehensive view of performance metrics as well as a systematic approach to loss minimisation which can be summarised as follows: given a model, apply the threshold choice methods that correspond with the available information about the operating condition, and compare their expected losses.