Predicting relative performance of classifiers from samples
Published 1 January 2005
Rui Leite, Pavel Brazdil
Citations59
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Experimental evaluation shows that the method competes well with previous approaches and provides quite good and practical solution to the problem of predicting relative performance of classification algorithms.
Abstract
This paper is concerned with the problem of predicting relative performance of classification algorithms. It focusses on methods that use results on small samples and discusses the shortcomings of previous approaches. A new variant is proposed that exploits, as some previous approaches, metalearning. The method requires that experiments be conducted on few samples. The information gathered is used to identify the nearest learning curve for which the sampling procedure was carried out fully. This in turn permits to generate a prediction regards the relative performance of algorithms. Experimental evaluation shows that the method competes well with previous approaches and provides quite good and practical solution to this problem. 1.
Keywords
Computer Science
ACM Transactions on Intelligent Systems and TechnologyLIBSVM
41,340 Citations2011Chih-Chung Chang, Chih‐Jen Lin
Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.
Cambridge University Press eBooksCase-Based Reasoning
4,254 Citations2005Janet L. Kolodner
It is argued that CBR systems are easy to assess, develop, and deploy and can be evaluated in terms of their cost benefits and argued that CBR systems are easy to assess, develop, and deploy.
Case-Based Reasoning: Experiences, Lessons and Future Directions
901 Citations1996David Leake
This book presents a selection of recent progress, issues, and directions for the future of case-based reasoning, and experimentally examines one of the fundamental tenets of CBR, that reasoning from prior experiences improves performance.
Machine LearningRanking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results
404 Citations2003Pavel Brazdil, Carlos Soares +1 more
A meta-learning method that uses a k-Nearest Neighbor algorithm to identify the datasets that are most similar to the one at hand and leads to significantly better rankings than the baseline ranking method.
ACM SIGART BulletinBook review: Machine Learning, Neural and Statistical Classification Edited by D. Michie, D.J. Spiegelhalter and C.C. Taylor (Ellis Horwood Limited, 1994)
371 Citations1996Huan Liu
This well-edited book provides an up-to-date review of different approaches to classification, compares their performance on a wide range of challenging datasets, and draws conclusions on their applicability to realistic industrial problems.
Efficient progressive sampling
353 Citations1999Foster Provost, David Jensen +1 more
This work analyzes methods for progressive sampling using progressively larger samples as long as model accuracy improves, and concludes that progressive sampling can be remarkably efficient.
Meta-Learning by Landmarking Various Learning Algorithms
302 Citations2000Bernhard Pfahringer, Hilan Bensusan +1 more
Experiments show that landmarking selects, with moderate but reasonable level of success, the best performing of a set of learning algorithms.
Static versus dynamic sampling for data mining
171 Citations1996George H. John, Pat Langley
The "Probably Close Enough" criterion is introduced to describe the desired properties of a sample and dynamic sampling methods, which take into account the mining tool being used and can thus give better samples are discussed.
Lecture notes in computer scienceEstimating the Predictive Accuracy of a Classifier
91 Citations2001Hilan Bensusan, Alexandros Kalousis
It is shown that it is possible to estimate classifier performance with a high degree of confidence and gain knowledge about the classifier through the regression models generated and exploit the results of the models to predict the ranking of the inducers.
Intelligent Data AnalysisNOEMON: Design, implementation and performance results of an intelligent assistant for classifier selection
88 Citations1999Alexandros Kalousis
Lecture notes in computer scienceDiscovering Task Neighbourhoods through Landmark Learning Performances
75 Citations2000Hilan Bensusan, Christophe Giraud-Carrier
Landmarking is a novel and promising metalearning approach to model selection that uses accuracy estimates from simple and efficient learners to describe tasks and subsequently construct meta-classifiers that predict which one of a set of more elaborate learning algorithms is appropriate for a given problem.
An Evaluation of Landmarking Variants
63 Citations2001Johannes Fürnkranz, Johann Petrak
This paper investigates relative landmarking, which tries to exploit the relative order of the landmark measures instead of their absolute value, and proposes to use subsampling estimates as a different way for efficiently obtaining landmarks.
Lecture notes in computer scienceSampling-Based Relative Landmarks: Systematically Test-Driving Algorithms before Choosing
40 Citations2001Carlos Soares, Johann Petrak +1 more
Sampling-based landmarks (SL), a systematization of this approach, building on earlier work on landmarking and sampling, are described, that address the inability of earlier landmarks to assess relative performance of algorithms.