An analysis of rule evaluation metrics
Published 21 August 2003
Johannes Fürnkranz, Peter Flach
Citations86
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
All commonly used heuristics, including accuracy, weighted relative accuracy, entropy, Gini index and information gain, are equivalent to one of two fundamental prototypes: precision and cost-weighted difference between covered positive and negative examples.
Abstract
In this paper we analyze the most popular evaluation metrics for separate-and-conquer rule learning algorithms. Our results show that all commonly used heuristics, including accuracy, weighted relative accuracy, entropy, Gini index and information gain, are equivalent to one of two fundamental prototypes: precision, which tries to optimize the area under the ROC curve for unknown costs, and a cost-weighted difference between covered positive and negative examples, which tries to find the optimal point under known or assumed costs. We also show that a straightforward generalization of the m-estimate trades off these two prototypes. 1.
Keywords
Computer Science
Machine LearningInduction of Decision Trees
14,815 Citations1986J. R. Quinlan
This paper summarizes an approach to synthesizing decision trees that has been used in a variety of systems, and it describes one such system, ID3, in detail, which is described in detail.
Elsevier eBooksFast Effective Rule Induction
3,767 Citations1995William W. Cohen
This paper evaluates the recently-proposed rule learning algorithm IREP on a large and diverse collection of benchmark problems, and proposes a number of modifications resulting in an algorithm RIPPERk that is very competitive with C4.5 and C 4.5rules with respect to error rates, but much more efficient on large samples.
Machine LearningThe CN2 Induction Algorithm
2,015 Citations1989Peter Clark, Tim Niblett
A description and empirical evaluation of a new induction system, CN2, designed for the efficient induction of simple, comprehensible production rules in domains where problems of poor description language and/or noise may be present.
Machine LearningLearning Logical Definitions from Relations
1,425 Citations1990J. R. Quinlan
FOIL is a system that learns Horn clauses from data expressed as relations, but extends them to a first-order formalism, which has been applied successfully to several tasks taken from the machine learning literature.
Machine LearningRobust Classification for Imprecise Environments
1,206 Citations2001Foster Provost, Tom Fawcett
It is shown that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions, and in some cases, the performance of the hybrid actually can surpass that of the best known classifier.
Induction of Decision Trees
999 Citations2003Quinlan
Artificial Intelligence ReviewSeparate-and-Conquer Rule Learning
523 Citations1999Johannes Fürnkranz
This paper is a survey of inductive rule learning algorithms that use a separate-and-conquer strategy and analyzes them along three different dimensions, namely their search, language and overfitting avoidance biases.
Lecture notes in computer scienceRule Evaluation Measures: A Unifying View
374 Citations1999Nada Lavrač, Peter Flach +1 more
This paper develops a unifying view on some of the existing measures for predictive and descriptive induction by means of contingency tables, and demonstrates that many rule evaluation measures developed for predictive knowledge discovery can be adapted to descriptive knowledge discovery tasks.
Learning Decision Trees Using the Area Under the ROC Curve
266 Citations2002Cèsar Ferri, Peter Flach +1 more
This paper shows how a single decision tree can represent a set of classifiers by choosing different labellings of its leaves, or equivalently, an ordering on the leaves, and proposes a novel splitting criterion which chooses the split with the highest local AUC.
Journal of Artificial Intelligence ResearchExpert-Guided Subgroup Discovery: Methodology and Application
204 Citations2002Dragan Gamberger, Nada Lavrač
The main step of the subgroup discovery process, the induction of subgroup descriptions, is performed by a heuristic beam search algorithm, using a novel parametrized definition of rule quality which is analyzed in detail.
Pattern Recognition LettersROC curves and the X2 test
24 Citations1996Andrew P. Bradley
An extension to the ROC curve is proposed so that it shows contours of X^2 values, which can be used to provide further insight into the appropriate setting of the decision threshold for a particular application.
A Quantification of Distance Bias Between Evaluation Metrics In Classification
20 Citations2000Ricardo Vilalta, Daniel Oblinger
An Analysis of Rule Learning Heuristics
7 Citations2003Johannes Fürnkranz, Peter Flach
All commonly used heuristics, including accuracy, weighted relative accuracy, entropy, Gini index and information gain, are equivalent to one of two fundamental prototypes: precision, which tries to optimize the area under the ROC curve for unknown costs, and a cost-eighted difference between covered positive and negative examples.