A support vector method for multivariate performance measures
Published 1 January 2005
Thorsten Joachims
Citations800
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An algorithm with which such multivariate SVMs can be trained in polynomial time for large classes of potentially non-linear performance measures, in particular ROCArea and all measures that can be computed from the contingency table are given.
Abstract
This paper presents a Support Vector Method for optimizing multivariate nonlinear performance measures like the F1-score. Taking a multivariate prediction approach, we give an algorithm with which such multivariate SVMs can be trained in polynomial time for large classes of potentially non-linear performance measures, in particular ROCArea and all measures that can be computed from the contingency table. The conventional classification SVM arises as a special case of our method.
Keywords
Computer Science
Machine LearningSupport-Vector Networks
33,035 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
A training algorithm for optimal margin classifiers
11,594 Citations1992Bernhard E. Boser, Isabelle Guyon +1 more
A training algorithm that maximizes the margin between the training patterns and the decision boundary is presented, applicable to a wide variety of the classification functions, including Perceptrons, polynomials, and Radial Basis Functions.
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
4,863 Citations1999John Platt
The output of a lassi(cid:12)er should be a alibrated posterior probability to enable post-pro essing and a method to train a kernel lassi with a logit link and a regularized maximum likelihood is proposed.
Support vector machine learning for interdependent and structured output spaces
1,260 Citations2004Ioannis Tsochantaridis, Thomas Hofmann +2 more
This paper proposes to generalize multiclass Support Vector Machine learning in a formulation that involves features extracted jointly from inputs and outputs, and demonstrates the versatility and effectiveness of the method on problems ranging from supervised grammar learning and named-entity recognition, to taxonomic text classification and sequence alignment.
Max-Margin Markov Networks
1,251 Citations2003Ben Taskar, Carlos Guestrin +1 more
Maximum margin Markov (M3) networks incorporate both kernels, which efficiently deal with high-dimensional features, and the ability to capture correlations in structured data, and a new theoretical bound for generalization in structured domains is provided.
AUC Optimization vs. Error Rate Minimization
512 Citations2003Corinna Cortes, Mehryar Mohri
The results show that the average AUC is monotonically increasing as a function of the classification accuracy, but that the standard deviation for uneven distributions and higher error rates is noticeable, so algorithms designed to minimize the error rate may not lead to the best possible AUC values.
Applied Physics Letters10.1162/1532443041827916
393 Citations2000
This work describes and analyze an efficient algorithm called RankBoost for combining preferences based on the boosting approach to machine learning, and gives theoretical results describing the algorithm's behavior both on the training data, and on new test data not seen during training.
A study of thresholding strategies for text categorization
354 Citations2001Yiming Yang
Experimental results show that the choice of thresholding strategy can significantly influence the performance of kNN, and that the ``optimal'' strategy may vary by application.
Econstor (Econstor)Combining statistical learning with a knowledge-based approach: A case study in intensive care monitoring
354 Citations1999Katharina Morik, Peter Brockhausen +1 more
Machine LearningSupport Vector Machines for Classification in Nonstandard Situations
348 Citations2002Yi Lin, Yoonkyung Lee +1 more
This paper explains why the standard support vectors machine is not suitable for the nonstandard situation, and introduces a simple procedure for adapting the support vector machine methodology to the non standard situation.
Evaluating and optimizing autonomous text classification systems
312 Citations1995David Lewis
This work shows how to define what constitutes good effectiveness for binary text classification systems, tune the systems to achieve the highest possible effectiveness, and estimate how the effectiveness changes as new data is processed.
Data mining in metric space
301 Citations2004Rich Caruana, Alexandru Niculescu-Mizil
A new metric is introduced, SAR, that combines squared error, accuracy, and ROC area into one metric, and MDS and correlation analysis shows that SAR is centrally located and correlates well with other metrics, suggesting that it is a good general purpose metric to use when more specific criteria are not known.
Learning Decision Trees Using the Area Under the ROC Curve
266 Citations2002Cèsar Ferri, Peter Flach +1 more
This paper shows how a single decision tree can represent a set of classifiers by choosing different labellings of its leaves, or equivalently, an ordering on the leaves, and proposes a novel splitting criterion which chooses the split with the highest local AUC.
Optimizing classifier performance via an approximation to the Wilcoxon-Mann-Whitney statistic
211 Citations2003Lian Yan, Robert H. Dodier +2 more
This work proposes an objective function that is an approximation to the Wilcoxon-Mann-Whitney statistic, which is equivalent to the AUC, and applies it to real-world customer behavior prediction problems for a wireless service provider and a cable service provider, and achieves reliable improvements in the ROC curve.
Optimising area under the ROC curve using gradient descent
196 Citations2004Alan Herschtal, Bhavani Raskutti
This paper introduces RankOpt, a linear binary classifier which optimises the area under the ROC curve (the AUC).
An iterative method for multi-class cost-sensitive learning
141 Citations2004Naoki Abe, Bianca Zadrozny +1 more
This paper empirically evaluates the performance of the proposed method using benchmark data sets and proves that the method generally achieves better results than representative methods for cost-sensitive learning, in terms of predictive performance (cost minimization) and, in many cases, computational efficiency.
The Florida AI Research SocietyOptimizing F-Measure with Support Vector Machines
81 Citations2003David R. Musicant, Vipin Kumar +1 more
It is demonstrated that with the right parameter settings SVMs approximately optimize F-measure in the same way that SVMs have already been known to approximately optimize accuracy.
Technische Universität Dortmund Eldorado (Technische Universität Dortmund)Combining statistical learning with a knowledge-based approach
13 Citations1999Peter Brockhausen, Thorsten Joachims +1 more
A case study in combining different methods for acquiring medical knowledge given a huge amount of noisy, high dimensional numerical time series data describing patients in intensive care, the support vector machine is used to learn when and how to change the dose of which drug.