New multicategory boosting algorithms based on multicategory Fisher-consistent losses
The Annals of Applied StatisticsPublished 1 December 2008Open access
Hui Zou, Ji Zhu, Trevor Hastie
Citations86
SJR quartileQ1
SJR score0.94
SNIP0.93
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper characterize a wide class of smooth convex loss functions that are Fisher-consistent for multicategory classification and derives two new multicategory boosting algorithms by using the exponential and logistic regression losses.
Abstract
Fisher-consistent loss functions play a fundamental role in the construction of successful binary margin-based classifiers. In this paper we establish the Fisher-consistency condition for multicategory classification problems. Our approach uses the margin vector concept which can be regarded as a multicategory generalization of the binary margin. We characterize a wide class of smooth convex loss functions that are Fisher-consistent for multicategory classification. We then consider using the margin-vector-based loss functions to derive multicategory boosting algorithms. In particular, we derive two new multicategory boosting algorithms by using the exponential and logistic regression losses.
Keywords
Computer ScienceEngineering
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
The Annals of StatisticsGreedy function approximation: A gradient boosting machine.
28,973 Citations2001Jerome H. Friedman
A general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion, and specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification.
Springer series in statisticsThe Elements of Statistical Learning
24,344 Citations2001Trevor Hastie, J. Friedman +1 more
Journal of Computer and System SciencesA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
20,320 Citations1997Yoav Freund, Robert E. Schapire
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
19,345 Citations2013Trevor Hastie, Robert Tibshirani +1 more
The Annals of StatisticsAdditive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)
6,842 Citations2000Jerome H. Friedman, Trevor Hastie +1 more
This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.
Improved boosting algorithms using confidence-rated predictions
2,568 Citations1998Robert E. Schapire, Yoram Singer
Machine LearningImproved Boosting Algorithms Using Confidence-rated Predictions
1,959 Citations1999Robert E. Schapire, Yoram Singer
Several improvements to Freund and Schapire's AdaBoost boosting algorithm are described, particularly in a setting in which hypotheses may assign confidences to each of their predictions.
In Defense of One-Vs-All Classification
1,387 Citations2004Ryan Rifkin, Aldebaro Klautau
It is argued that a simple "one-vs-all" scheme is as accurate as any other approach, assuming that the underlying binary classifiers are well-tuned regularized classifiers such as support vector machines.
Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers
914 Citations2000Erin L. Allwein, Robert E. Schapire +1 more
A general method for combining the classifiers generated on the binary problems is proposed, and a general empirical multiclass loss bound is proved given the empirical loss of the individual binary learning algorithms.
Journal of the American Statistical AssociationBoosting With the<i>L</i><sub>2</sub>Loss
864 Citations2003Peter Bühlmann, Bin Yu
It is demonstrated that L2Boosting with a novel component-wise cubic smoothing spline is both practical and effective in the presence of high-dimensional predictors.
Journal of the American Statistical AssociationMulticategory Support Vector Machines
747 Citations2004Yoonkyung Lee, Yi Lin +1 more
Statistical ScienceBoosting Algorithms: Regularization, Prediction and Model Fitting
720 Citations2007Peter Bühlmann, Torsten Hothorn
A statistical perspective on boosting is presented, with special emphasis on estimating potentially complex parametric or nonparametric models, including generalized linear and additive models as well as regression models for survival analysis.
The Annals of StatisticsStatistical behavior and consistency of classification methods based on convex risk minimization
709 Citations2004Tong Zhang
This study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines and shows their limitations and suggests possible improvements.
The Annals of StatisticsEmpirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers
468 Citations2002Vladimir Koltchinskii, Dmitry Panchenko
New probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifier combinations, based on the methods of the theory of Gaussian and empirical processes are proved.
Data Mining and Knowledge DiscoverySupport Vector Machines and the Bayes Rule in Classification
228 Citations2002Yi Lin
It is shown that the asymptotic target of SVMs are some interesting classification functions that are directly related to the Bayes rule, and helps understand the success of SVM in many classification studies, and makes it easier to compare SVMs and traditional statistical methods.
Statistical Analysis of Some Multi-Category Large Margin Classification Methods
215 Citations2004Tong Zhang
Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications
209 Citations2005Andreas Buja, Werner Stuetzle +1 more
The use of proper scoring rules with novel criteria for 1) Hand and Vinciotti’s (2003) localized logistic regression and 2) for interpretable classification trees are illustrated.
The Annals of StatisticsOn the Bayes-risk consistency of regularized boosting methods
188 Citations2004Gábor Lugosi, Nicolas Vayatis
The main result of the paper is that certain regularized boosting algorithms provide Bayes-risk consistent classifiers under the sole assumption that the Bayes classifier may be approximated by a convex combination of the base classifiers.
Statistics & Probability LettersA note on margin-based loss functions in classification
178 Citations2004Yi Lin
It is shown that a large family of margin-based loss functions are Fisher consistent for classification, that is, the population minimizer of the loss function leads to the Bayes optimal rule of classification.
Journal of the American Statistical AssociationMulticategory <i>ψ</i>-Learning
139 Citations2006Yufeng Liu, Xiaotong Shen
A novel multicategory generalization of ψ-learning that treats all classes simultaneously and can deliver accurate class prediction and is more robust against extreme observations than its SVM counterpart.
Journal of Computational and Graphical StatisticsMulticategory ψ-Learning and Support Vector Machine: Computational Tools
109 Citations2005Yufeng Liu, Xiaotong Shen +1 more
The first computational tools for multicategory ψ-learning are based on d.c. algorithms and solved by sequential quadratic programming, while the second one uses the outer approximation method, which yields the global minimizer via sequential concave minimization.
Journal of Machine Learning ResearchOn the rate of convergence of regularized boosting classifiers
109 Citations2003Gilles Blanchard, Gábor Lugosi +1 more
It is shown that some versions of boosting work especially well in high-dimensional logistic additive models, and adding a limited labelling noise to the training data may in certain cases improve the convergence.