A novel LS-SVMs hyper-parameter selection based on particle swarm optimization
NeurocomputingPublished 23 June 2008
Xinchen Guo, Jianbo Yang, Chunguo Wu, Cong Wang, Yanchun Liang
Citations210
SJR quartileQ1
SJR score1.47
SNIP1.94
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel hyper-parameter selection method for LS-SVMs is presented based on the particle swarm optimization (PSO), which does not need any priori knowledge on the analytic property of the generalization performance measure and can be used to determine multiplehyper-parameters at the same time.
Abstract
The selection of hyper-parameters plays an important role to the performance of least-squares support vector machines (LS-SVMs). In this paper, a novel hyper-parameter selection method for LS-SVMs is presented based on the particle swarm optimization (PSO). The proposed method does not need any priori knowledge on the analytic property of the generalization performance measure and can be used to determine multiple hyper-parameters at the same time. The feasibility of this method is examined on benchmark data sets. Different kinds of kernel families are investigated by using the proposed method. Experimental results show that the best or quasi-best test performance could be obtained by using the scaling radial basis kernel function (SRBF) and RBF kernel functions, respectively.
Keywords
ChemistryDecision SciencesEngineering
Particle swarm optimization
47,523 Citations2002James Kennedy, R.C. Eberhart
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
Swarm IntelligenceParticle swarm optimization
21,351 Citations2007Riccardo Poli, James Kennedy +1 more
A snapshot of particle swarming from the authors’ perspective, including variations in the algorithm, current and ongoing research, applications and open problems, is included.
Cambridge University Press eBooksAn Introduction to Support Vector Machines and Other Kernel-based Learning Methods
13,883 Citations2000Nello Cristianini, John Shawe‐Taylor
A modified particle swarm optimizer
10,157 Citations2002Yibing Shi, R.C. Eberhart
A new parameter, called inertia weight, is introduced into the original particle swarm optimizer, which resembles a school of flying birds since it adjusts its flying according to its own flying experience and its companions' flying experience.
Neural Processing LettersLeast Squares Support Vector Machine Classifiers
9,375 Citations1999Johan A. K. Suykens, Joos Vandewalle
A least squares version for support vector machine (SVM) classifiers that follows from solving a set of linear equations, instead of quadratic programming for classical SVM's.
IEEE Transactions on Evolutionary ComputationThe particle swarm - explosion, stability, and convergence in a multidimensional complex space
8,768 Citations2002Maurice Clerc, James Kennedy
Machine LearningChoosing Multiple Parameters for Support Vector Machines
2,182 Citations2002Olivier Chapelle, Vladimir Vapnik +2 more
The problem of automatically tuning multiple parameters for pattern recognition Support Vector Machines (SVMs) is considered by minimizing some estimates of the generalization error of SVMs using a gradient descent algorithm over the set of parameters.
Machine LearningSoft Margins for AdaBoost
1,306 Citations2001Gunnar Rätsch, Takashi Onoda +1 more
It is found that ADABOOST asymptotically achieves a hard margin distribution, i.e. the algorithm concentrates its resources on a few hard-to-learn patterns that are interestingly very similar to Support Vectors.
Machine LearningBenchmarking Least Squares Support Vector Machine Classifiers
749 Citations2003Tony Van Gestel, Johan A. K. Suykens +6 more
Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances, consistently very good when compared to a variety of methods described in the literature.
Neural ComputationBounds on Error Expectation for Support Vector Machines
627 Citations2000Vladimir Vapnik, Olivier Chapelle
It is proved that the value of the span is always smaller (and can be much smaller) than the diameter of the smallest sphere containing the support vectors, used in previous bounds.
NeurocomputingEvolutionary tuning of multiple SVM parameters
506 Citations2005Frauke Friedrichs, Christian Igel
It is demonstrated on benchmark datasets that the CMA-ES improves the results achieved by grid search already when applied to few hyperparameters and that tuning of the scaling and the rotation of Gaussian kernel can lead to better results in comparison to standard Gaussian kernels with a single bandwidth parameter.
Energy Conversion and ManagementSupport vector machines with simulated annealing algorithms in electricity load forecasting
382 Citations2005Ping‐Feng Pai, Wei‐Chiang Hong
The empirical results reveal that the proposed SVMSA model outperforms the other two models, namely the autoregressive integrated moving average (ARIMA) model and the general regression neural networks (GRNN) model, and provides a promising alternative for forecasting electricity load.
Model Selection for Support Vector Machines
347 Citations1999Olivier Chapelle, Vladimir Vapnik
New functionals for parameter (model) selection of Support Vector Machines are introduced based on the concepts of the span of support vectors and rescaling of the feature space and it is shown that using these functionals one can both predict the best choice of parameters of the model and the relative quality of performance for any value of parameter.
Feature selection for support vector machines by means of genetic algorithm
338 Citations2004Holger Fröhlich, Olivier Chapelle +1 more
This paper presents a special genetic algorithm, which especially takes into account the existing bounds on the generalization error for support vector machines (SVMs), which is compared to the traditional method of performing cross-validation and to other existing algorithms for feature selection.
IEEE Transactions on Neural NetworksEfficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms
315 Citations2002S. Sathiya Keerthi
The paper discusses implementation issues related to the tuning of the hyperparameters of a support vector machine (SVM) with L/sub 2/ soft margin, for which the radius/margin bound is taken as the index to be minimized, and iterative techniques are employed for computing radius and margin.
Sparse approximation using least squares support vector machines
315 Citations2002Johan A. K. Suykens, L. Lukas +1 more
This paper investigates imposing sparseness by pruning support values from the sorted support value spectrum which results from the solution to the linear system.
Pattern RecognitionFast cross-validation algorithms for least squares support vector machine and kernel ridge regression
300 Citations2007Senjian An, Wanquan Liu +1 more
A novel method for cross-validation (CV) of LS-SVM or KRR is proposed in which instead of solving l linear systems of dimension (n-m), the inverse of an n dimensional square matrix is computed and solved to reduce the complexity when l is large and/or m is small.
Neural ComputationRadius Margin Bounds for Support Vector Machines with the RBF Kernel
219 Citations2003Kai-Min Chung, Wei‐Chun Kao +3 more
NeurocomputingModel selection for support vector machine classification
201 Citations2003Carl Gold, Peter Sollich
The results for the evidence gradient ascent method show that also the exact evidence exhibits local optima, but these give test errors which are much less variable and also consistently lower than for the simpler model selection criteria.
Pattern RecognitionAutomatic model selection for the optimization of SVM kernels
181 Citations2005N.E. Ayat, Mohamed Cheriet +1 more
The experiments conducted on a bi-class problem show that the proposed methodology can adequately choose the SVM hyper-parameters using the empirical error criterion and it turns out that the criterion produces a less complex model with fewer support vectors.
Applied Mathematics and ComputationDetection of ECG Arrhythmia using a differential expert system approach based on principal component analysis and least square support vector machine
142 Citations2006Kemal Polat, Salih Güneş
This result is for ECG Arrhythmias disease but it states that this method can be used confidently for other medical diseases diagnosis problems, too.
Applied Soft ComputingText classification: A least square support vector machine approach
140 Citations2006Vikramjit Mitra, Chia-Jiu Wang +1 more
The comparison between the conventional linear SVM based classifiers and neural network based classifying agents shows that the LS-SVM with LSI basedclassifying agents improves text categorization performance significantly and holds a lot of potential for developing robust learning based agents for text classification.
Pattern Recognition LettersEfficient computations for large least square support vector machine classifiers
110 Citations2002Kok Seng Chua
The linear system in the training of the least square support vector machine (LSSVM) proposed by Suykens and Vandewalle can be placed in a more symmetric form so that for a data set with N data points and m features, the linear system can be solved by inverting an m × m instead of an N × N matrix and storing and working with matrices of size at most m × N.
Neural ComputationLeave-One-Out Bounds for Support Vector Regression Model Selection
108 Citations2005Ming‐Wei Chang, Chih‐Jen Lin
Experiments demonstrate that the proposed bounds are competitive with Bayesian SVR for parameter selection and the differentiability of leave-one-out bounds is discussed.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIEGenetic Algorithms and Support Vector Machines for Time Series Classification
107 Citations2002Damian Eads, Daniel Hill +5 more
This work introduces an algorithm for classifying time series data that employs evolutionary computation for feature extraction, and a support vector machine for the final backend classification.
IEEE Transactions on Geoscience and Remote SensingSemisupervised PSO-SVM Regression for Biophysical Parameter Estimation
86 Citations2007Yakoub Bazi, Farid Melgani
A novel semisupervised regression approach is proposed to tackle the problem of biophysical parameter estimation that is constrained by a limited availability of training (labeled) samples by exploiting unlabeled samples that are available from the image under analysis at zero cost.
Neural Computing and ApplicationsClassification of ECG beats by using a fast least square support vector machines with a dynamic programming feature selection algorithm
81 Citations2005Nurettin Acır
Experimental results show that not only the fast L SSVM is faster than the standard LSSVM algorithm, but also it gives better classification performance than thestandard backpropagation multilayer perceptron network.
Fuel and Energy Abstracts05/02577 Support vector machines with simulated annealing algorithms in electricity load forecasting
75 Citations2005
Molecular PharmaceuticsQSAR and Classification Study of 1,4-Dihydropyridine Calcium Channel Antagonists Based on Least Squares Support Vector Machines
59 Citations2005Xiaojun Yao, Huanxiang Liu +6 more
A new and effective method for drug design and screening for a novel series of 1,4-dihydropyridine calcium channel antagonists for the first time using the least squares support vector machine.
Neural ComputationGradient-Based Adaptation of General Gaussian Kernels
58 Citations2005Tobias Glasmachers, Christian Igel
Gradient-based optimizing of gaussian kernel functions is considered, and the gradient for the adaptation of scaling and rotation of the input space is computed to achieve invariance against linear transformations.
Journal of Chemical Information and ModelingRadial Basis Function Network-Based Transform for a Nonlinear Support Vector Machine as Optimized by a Particle Swarm Optimization Algorithm with Application to QSAR Studies
49 Citations2007Li‐Juan Tang, Yanping Zhou +5 more
Applications of the proposed algorithm to QSAR studies of binding affinity of HIV-1 reverse transcriptase inhibitors and activity of 1-phenylbenzimidazoles reveal that the new algorithm provides superior performance to the backpropagation neural network and a conventional non linear SVM, indicating that this algorithm holds great promise in nonlinear SVM learning.
The Bias-Variance Tradeoff and the Randomized GACV
41 Citations1998Grace Wahba, Xiwu Lin +4 more
Amino AcidsPrediction of mitochondrial proteins based on genetic algorithm – partial least squares and support vector machine
37 Citations2007Fengxiao Tan, Xiang Feng +4 more
This study used the genetic algorithm-partial least square (GA-PLS) method to evaluate the dipeptide composition elements which are more important in recognizing mitochondrial proteins, and selected these selected elements were applied to support vector machine (SVM)-based classifiers to predict the mitochondrial proteins.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIEFeature selection for computer-aided polyp detection using genetic algorithms
37 Citations2003Meghan T. Miller, Anna Jerebko +2 more
It is concluded that genetic algorithms are effective in comparison to the forward search procedure when used in conjunction with a committee of support vector machine classifiers for the purpose of colonic polyp identification.
Chemometrics and Intelligent Laboratory SystemsQuantitative structure–activity relationship study of acyl ureas as inhibitors of human liver glycogen phosphorylase using least squares support vector machines
31 Citations2007Jiazhong Li, Huanxiang Liu +4 more
The prediction results indicate that LS-SVMs is a potential method in QSAR study and can be used as a tool of drug screening.
International Journal of Computational MethodsA PARTICLE SWARM OPTIMIZATION-BASED ALGORITHM FOR JOB-SHOP SCHEDULING PROBLEMS
30 Citations2005Hongwei Ge, Yanchun Liang +2 more
A novel particle swarm optimization (PSO)-based algorithm is developed for job-shop scheduling problems (JSSP), which are the most general and difficult issues in traditional scheduling problems.
Applied Mathematics and ComputationA novel approach to estimation of E. coli promoter gene sequences: Combining feature selection and least square support vector machine (FS_LSSVM)
30 Citations2007Kemal Polat, Salih Güneş
Results indicate that the proposed approach improve the success rate in recognizing promoters in strings that represent nucleotides by means of combining feature selection and least square support vector machine.
Journal of Agricultural and Food ChemistryUse of Visible and Near Infrared Spectroscopy and Least Squares-Support Vector Machine To Determine Soluble Solids Content and pH of Cola Beverage
29 Citations2007Fei Liu, Yong He
The results indicated that VIS/NIR spectroscopy combined with LS-SVM could be applied as a rapid and alternative way for the fast determination of SSC and pH of cola beverage.
Annals of Biomedical EngineeringPattern Detection of Atherosclerosis from Carotid Artery Doppler Signals using Fuzzy Weighted Pre-Processing and Least Square Support Vector Machine (LSSVM)
22 Citations2007Kemal Polat, Sadık Kara +2 more
The employed expert system has achieved 100% classification accuracy using a 10-fold Cross Validation (CV) method and fuzzy weighted pre-processing based is proposed expert system, applied to inputs obtained from spectral analysis of Carotid Artery Doppler Signals.
Lecture notes in computer scienceAnalysis of Proteomic Pattern Data for Cancer Detection
22 Citations2004Kees H. de Jong, Elena Marchiori +1 more
Two proteomic pattern datasets containing measurements from ovarian and prostate cancer samples are analyzed by means of a linear and a quadratic support vector machine (SVM) and an evolutionary algorithm for feature selection (EAFS) that searches for small subsets of features in order to optimize the SVM performance.
Discriminative Densities from Maximum Contrast Estimation
18 Citations2002Peter Meinicke, Thorsten Twellmann +1 more
The close relation between SVM- and MCC-training is indicated and it is shown that Linear Programming Machines can be viewed as an approximate realization of MCCs, which have the same functional form as the well-known Support Vector Machines.
Optical EngineeringNovel algorithm for image interpolation
13 Citations2004Jinwen Tian
A novel interpolation algorithm to find the optimal image intensity function generating an optimal gray-level estimation of interpolated pixels of digital images based on the proposed image block mapping method and least-square support vector machines with Gaussian radial basis function (RBF) kernels is described.
Feature Selection for Support Vector Machines by Means of Genetic Algorithms
6 Citations2002Holger Fröhlich, Olivier Chapelle +1 more
Journal of Zhejiang University. Science ASVD-LSSVM and its application in chemical pattern classification
5 Citations2006Shaohui Tao, Dezhao Chen +1 more
SVD-LSSVM has good performance in classification and saves time for cross validation, and a new hyper parameter, singular value contribution rate (SVCR), replaces the penalty factor of LSSVM.
浙江大学学报：A卷英文版SVD-LSSVM and its application in chemical pattern classification
4 Citations2006Tao -, Shao-hui +4 more