Model selection for the LS-SVM. Application to handwriting recognition
Pattern RecognitionPublished 8 November 2008
Mathias M. Adankon, Mohamed Cheriet
Citations196
SJR quartileQ1
SJR score2.06
SNIP2.67
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper proposes to conduct model selection for the LS-SVM using an empirical error criterion and shows the usefulness of this classifier and demonstrates that model selection improves the generalization performance of the LS, the least squares SVM.
Abstract
The support vector machine (SVM) is a powerful classifier which has been used successfully in many pattern recognition problems. It has also been shown to perform well in the handwriting recognition field. The least squares SVM (LS-SVM), like the SVM, is based on the margin-maximization principle performing structural risk minimization. However, it is easier to train than the SVM, as it requires only the solution to a convex linear problem, and not a quadratic problem as in the SVM. In this paper, we propose to conduct model selection for the LS-SVM using an empirical error criterion. Experiments on handwritten character recognition show the usefulness of this classifier and demonstrate that model selection improves the generalization performance of the LS-SVM.
Keywords
Computer ScienceEngineering
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
Choice Reviews OnlineNeural networks for pattern recognition
18,687 Citations1994
This is the first comprehensive treatment of feed-forward neural networks from the perspective of statistical pattern recognition, and is designed as a text, with over 100 exercises, to benefit anyone involved in the fields of neural computation and pattern recognition.
Cambridge University Press eBooksAn Introduction to Support Vector Machines and Other Kernel-based Learning Methods
13,883 Citations2000Nello Cristianini, John Shawe‐Taylor
Neural Networks for Pattern Recognition
12,198 Citations1995Chris Bishop
A training algorithm for optimal margin classifiers
11,594 Citations1992Bernhard E. Boser, Isabelle Guyon +1 more
A training algorithm that maximizes the margin between the training patterns and the decision boundary is presented, applicable to a wide variety of the classification functions, including Perceptrons, polynomials, and Radial Basis Functions.
Neural Processing LettersLeast Squares Support Vector Machine Classifiers
9,375 Citations1999Johan A. K. Suykens, Joos Vandewalle
A least squares version for support vector machine (SVM) classifiers that follows from solving a set of linear equations, instead of quadratic programming for classical SVM's.
Cambridge University Press eBooksKernel Methods for Pattern Analysis
6,598 Citations2004John Shawe‐Taylor, Nello Cristianini
This book provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating with examples how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so.
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
4,863 Citations1999John Platt
The output of a lassi(cid:12)er should be a alibrated posterior probability to enable post-pro essing and a method to train a kernel lassi with a logit link and a regularized maximum likelihood is proposed.
ePrints Soton (University of Southampton)An Introduction to Support Vector Machines
4,120 Citations2000Nello Cristianini, John Shawe‐Taylor
WORLD SCIENTIFIC eBooksLeast Squares Support Vector Machines
3,640 Citations2002Johan A. K. Suykens, Tony Van Gestel +3 more
Machine LearningChoosing Multiple Parameters for Support Vector Machines
2,182 Citations2002Olivier Chapelle, Vladimir Vapnik +2 more
The problem of automatically tuning multiple parameters for pattern recognition Support Vector Machines (SVMs) is considered by minimizing some estimates of the generalization error of SVMs using a gradient descent algorithm over the set of parameters.
The MIT Press eBooksAdvances in Large-Margin Classifiers
1,870 Citations2000
This book provides an overview of recent developments in large margin classifiers, examines connections with other methods, and identifies strengths and weaknesses of the method, as well as directions for future research.
Machine LearningBenchmarking Least Squares Support Vector Machine Classifiers
749 Citations2003Tony Van Gestel, Johan A. K. Suykens +6 more
Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances, consistently very good when compared to a variety of methods described in the literature.
NeurocomputingEvaluation of simple performance measures for tuning SVM hyperparameters
634 Citations2003Kai-Bo Duan, S. Sathiya Keerthi +1 more
The empirically study the usefulness of several simple performance measures that are inexpensive to compute (in the sense that they do not require expensive matrix operations involving the kernel matrix) for tuning SVM hyperparameters.
Neural Information Processing SystemsPrinciples of Risk Minimization for Learning Theory
629 Citations1991Vladimir Vapnik
Systematic improvements in prediction power and empirical risk minimization are illustrated in application to zip-code recognition.
Neural ComputationBounds on Error Expectation for Support Vector Machines
627 Citations2000Vladimir Vapnik, Olivier Chapelle
It is proved that the value of the span is always smaller (and can be much smaller) than the diameter of the smallest sphere containing the support vectors, used in previous bounds.
Neural ComputationGradient-Based Optimization of Hyperparameters
478 Citations2000Yoshua Bengio
This article presents a methodology to optimize several hyper-parameters, based on the computation of the gradient of a model selection criterion with respect to the hyperparameter gradient involving second derivatives of the training criterion.
Model Selection for Support Vector Machines
347 Citations1999Olivier Chapelle, Vladimir Vapnik
New functionals for parameter (model) selection of Support Vector Machines are introduced based on the concepts of the span of support vectors and rescaling of the feature space and it is shown that using these functionals one can both predict the best choice of parameters of the model and the relative quality of performance for any value of parameter.
Technische Universität Dortmund Eldorado (Technische Universität Dortmund)Estimating the generalization performance of a SVM efficiently
328 Citations1999Thorsten Joachims
Without any computation-intensive resampling, the new estimators developed here are computationally much more e cient than cross-validation or bootstrapping and address the special performancemeasures needed for evaluating text classi ers.
Neural NetworksFast exact leave-one-out cross-validation of sparse least-squares support vector machines
324 Citations2004Gavin C. Cawley, Nicola L. C. Talbot
Neural ComputationBayesian Framework for Least-Squares Support Vector Machine Classifiers, Gaussian Processes, and Kernel Fisher Discriminant Analysis
293 Citations2002Tony Van Gestel, Johan A. K. Suykens +4 more
The LS-SVM formulation has clear primal-dual interpretations, and without the bias term, one explicitly constructs a model that yields the same expressions as have been obtained with GPs for regression, which has advantages for deriving analytic expressions in a Bayesian evidence framework.
Journal of Machine Learning ResearchPreventing Over-Fitting during Model Selection via Bayesian Regularisation of the Hyper-Parameters
227 Citations2007Gavin C. Cawley, Nicola L. C. Talbot
This paper investigates the novel use of Bayesian regularisation at the second level of inference, adding a regularisation term to the model selection criterion corresponding to a prior over the hyper-parameter values, where the additional regularisation parameters are integrated out analytically.
Neural ComputationRadius Margin Bounds for Support Vector Machines with the RBF Kernel
219 Citations2003Kai-Min Chung, Wei‐Chun Kao +3 more
Probabilistic Kernel Regression Models
207 Citations1999Tommi Jaakkola, David Haussler
A class of exible conditional probability models and techniques for classi cation regression problems that comes from the use of kernel functions as in support vector machines and the generality from dual formulations of stan dard regression models is introduced.
The 2006 IEEE International Joint Conference on Neural Network ProceedingsLeave-One-Out Cross-Validation Based Model Selection Criteria for Weighted LS-SVMs
197 Citations2006Gavin C. Cawley
A simple and efficient approach to model selection for weighted least-squares support vector machines, and compares a variety of model selection criteria based on leave-one-out cross-validation.
IEEE Transactions on Neural NetworksPruning error minimization in least squares support vector machines
193 Citations2003Bas J. de Kruif, Theo J.A. de Vries
A procedure is introduced that selects from a data set the training sample that will introduce the smallest approximation error when it will be omitted, and it is shown that this pruning scheme outperforms the standard one.
IEEE Transactions on Neural NetworksFast Sparse Approximation for Least Squares Support Vector Machine
190 Citations2007Licheng Jiao, Liefeng Bo +1 more
Two fast sparse approximation schemes for least squares support vector machine (LS-SVM) are presented to overcome the limitation of LS-S VM that it is not applicable to large data sets and to improve test speed.
Pattern RecognitionAutomatic model selection for the optimization of SVM kernels
181 Citations2005N.E. Ayat, Mohamed Cheriet +1 more
The experiments conducted on a bi-class problem show that the proposed methodology can adequately choose the SVM hyper-parameters using the empirical error criterion and it turns out that the criterion produces a less complex model with fewer support vectors.
Sparse least squares Support Vector Machine classifiers.
136 Citations2000Johan A. K. Suykens, Lukas Lukas +1 more
This paper proposes a method for imposing sparseness to the LS-SVM solution by pruning the support value spectrum which is revealing the relative importance of the training data points and is immediately available as solution to the linear systems.
Lecture notes in computer scienceImproved pairwise coupling classification with correcting classifiers
129 Citations1998Miguel Moreira, Eddy Mayoraz
Various ways of recombining the outputs of all the classifiers solving the existing subproblems are explored, and an important handicap of its intrinsic nature is exposed, which consists in the use, for the classification, of impertinent information.
NeurocomputingFeature vector selection and projection using kernels
117 Citations2003G. Baudat, F. Anouar
The kernel trick is used to select from the data a relevant subset forming a basis in a feature space F, and it will turn out that the size of the basis is related to the complexity of the model.
Pattern Recognition LettersEfficient computations for large least square support vector machine classifiers
110 Citations2002Kok Seng Chua
The linear system in the training of the least square support vector machine (LSSVM) proposed by Suykens and Vandewalle can be placed in a more symmetric form so that for a data set with N data points and m features, the linear system can be solved by inverting an m × m instead of an N × N matrix and storing and working with matrices of size at most m × N.
IEEE Transactions on Neural NetworksAn Improved Conjugate Gradient Scheme to the Solution of Least Squares SVM
95 Citations2005Wei Chu, C.J. Ong +1 more
An improved method to the numerical solution of LS-SVM is proposed and it is shown that the problem can be solved using one reduced system of linear equations.
NeurocomputingImproved sparse least-squares support vector machines
72 Citations2002Gavin C. Cawley, Nicola L. C. Talbot
An improved method for achieving sparsity in least-squares support vector machines, which takes into account the residuals for all training patterns, rather than only those incorporated in the sparse kernel expansion.
NeurocomputingImproved sparse least-squares support vector machine classifiers
47 Citations2006Yuangui Li, Chenxiao Lin +1 more
A different reduced training set is selected to re-train LS-SVM and a new procedure is proposed to obtain the sparseness and the results indicate that it is more effective.
Pattern RecognitionOptimizing resources in model selection for support vector machine
41 Citations2006Mathias M. Adankon, Mohamed Cheriet
This paper proposes a fast method based on an approximation of the gradient of the empirical error, along with incremental learning, which reduces the resources required both in terms of processing time and of storage space.
Gaussian Processes and SVM: Mean Field and Leave-One-Out
37 Citations2000Alexander J. Smola, Peter L. Bartlett +2 more
This chapter contains sections titled: Introduction, Gaussian Process Classification, Modeling the Noise, From Gaussian processes to SVM, Leave-One-Out Estimator, Naive Mean Field Algorithm, Simulation Results, and Conclusion.
Gradient Based Optimization of Hyper Parameters
20 Citations2007Yoshua Bengio
Aston Publications Explorer (Aston University)Mean Field Methods for Classification with Gaussian Processes
19 Citations1998Manfred Opper, Ole Winther
This work discusses the application of TAP mean field methods known from the Statistical Mechanics of disordered systems to Bayesian classification models with Gaussian processes, and results for the Sonar data set are given.
Aston Publications Explorer (Aston University)Advances in large margin classifiers
12 Citations2000Rainer Dietrich, Manfred Opper +1 more
Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.Optimizing resources in model selection for support vector machines
10 Citations2006Mathias M. Adankon, M. Cheriel +1 more
This paper proposes a fast method based on an approximation of the gradient of the empirical error along with incremental learning, which reduces the resources required both in terms of processing time and of storage space.
Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.Fast Bayesian support vector machine parameter tuning with the nystrom method
10 Citations2006Carl A. Gold, Peter Sollich
The Nystrom approximation to the SVM kernel significantly reduces the dimensionality of the space to be simulated in the HMC, and gives excellent performance in comparison with other approaches to hyperparameter tuning.