Training Invariant Support Vector Machines
Machine LearningPublished 1 January 2002Open access
Dennis DeCoste, Bernhard Schölkopf
Citations582
SJR quartileQ1
SJR score1.15
SNIP2.14
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Practical experience has shown that in order to obtain the best possible performance, prior knowledge about invariances of a classification problem at hand ought to be incorporated into the training procedure. We describe and review all known methods for doing so in support vector machines, provide experimental results, and discuss their respective merits. One of the significant new results reported in this work is our recent achievement of the lowest reported test error on the well-known MNIST digit recognition benchmark task, with SVM training times that are also significantly faster than previous SVM methods.
Keywords
Computer Science
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
Machine LearningSupport-vector networks
40,463 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
Machine LearningSupport-Vector Networks
33,035 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
TechnometricsStatistical Learning Theory
26,913 Citations1999Yuhai Wu, Vladimir Vapnik
Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
Neural ComputationBackpropagation Applied to Handwritten Zip Code Recognition
11,878 Citations1989Yann LeCun, Bernhard E. Boser +5 more
This paper demonstrates how constraints from the task domain can be integrated into a backpropagation network through the architecture of the network, successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service.
A training algorithm for optimal margin classifiers
11,594 Citations1992Bernhard E. Boser, Isabelle Guyon +1 more
A training algorithm that maximizes the margin between the training patterns and the decision boundary is presented, applicable to a wide variety of the classification functions, including Perceptrons, polynomials, and Radial Basis Functions.
Neural ComputationNonlinear Component Analysis as a Kernel Eigenvalue Problem
8,078 Citations1998Bernhard Schölkopf, Alexander J. Smola +1 more
A new method for performing a nonlinear form of principal component analysis by the use of integral operator kernel functions is proposed and experimental results on polynomial feature extraction for pattern recognition are presented.
International Conference on Neural Information ProcessingAdvances in kernel methods: support vector learning
5,815 Citations1999Bernhard Schölkopf, Christopher J. C. Burges +1 more
Support vector machines for dynamic reconstruction of a chaotic system, Klaus-Robert Muller et al pairwise classification and support vector machines, Ulrich Kressel.
The MIT Press eBooksFast Training of Support Vector Machines Using Sequential Minimal Optimization
5,462 Citations1998John Platt
The MIT Press eBooksAdvances in Large-Margin Classifiers
1,870 Citations2000
This book provides an overview of recent developments in large margin classifiers, examines connections with other methods, and identifies strengths and weaknesses of the method, as well as directions for future research.
Neural ComputationImprovements to Platt's SMO Algorithm for SVM Classifier Design
1,817 Citations2001S. Sathiya Keerthi, Shirish Shevade +2 more
Using clues from the KKT conditions for the dual problem, two threshold parameters are employed to derive modifications of SMO that perform significantly faster than the original SMO on all benchmark data sets tried.
Exploiting Generative Models in Discriminative Classifiers
1,285 Citations1998Tommi Jaakkola, David Haussler
A natural way of achieving this combination by deriving kernel functions for use in discriminative methods such as support vector machines from generative probability models is developed.
The Annals of Mathematical StatisticsA Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines
950 Citations1970George Kimeldorf, Grace Wahba
A Theory of Networks for Approximation and Learning
716 Citations1989Tomaso Poggio, Federico Girosi
Neural NetworksThe connection between regularization operators and support vector kernels
621 Citations1998Alex Smola, Bernhard Schölkopf +1 more
It is proved that the Green's Functions associated with regularization operators are suitable support vector kernels with equivalent regularization properties and it is shown that a large number of radial basis functions, namely conditionally positive definite functions, may be used as support vector kernel.
Comparison of learning algorithms for handwritten digit recognition
584 Citations1995Yann LeCun, L. D. Jackel +10 more
This comparison of several learning algorithms for handwritten digits considers not only raw accuracy, but also rejection, training time, recognition time, and memory requirements.
Neural ComputationLocal Learning Algorithms
544 Citations1992Léon Bottou, Vladimir Vapnik
A single analysis suggests that neither kNN or RBF, nor nonlocal classifiers, achieve the best compromise between locality and capacity.
Extracting support data for a given task
542 Citations1995Bernhard Schölkopf, Chris Burges +1 more
It is observed that three different types of handwritten digit classifiers construct their decision surface from strongly overlapping small subsets of the data base, which opens up the possibility of compressing data bases significantly by disposing of theData which is not important for the solution of a given task.
Neural ComputationAn Equivalence Between Sparse Approximation and Support Vector Machines
505 Citations1998Federico Girosi
If the data are noiseless, the modified version of basis pursuit denoising proposed in this article is equivalent to SVM in the following sense: if applied to the same data set, the two techniques give the same solution, which is obtained by solving the same quadratic programming problem.
Neural Information Processing SystemsEfficient Pattern Recognition Using a New Transformation Distance
464 Citations1992Patrice Simard, Yann LeCun +1 more
A new distance measure which can be made locally invariant to any set of transformations of the input and can be computed efficiently is proposed.
BioinformaticsEngineering support vector machine kernels that recognize translation initiation sites
425 Citations2000Alexander Zien, Gunnar Rätsch +4 more
With the described techniques the recognition performance can be improved by 26% over leading existing approaches, and there is evidence that existing related methods could profit from advanced TIS recognition.
Prior Knowledge in Support Vector Kernels
293 Citations1997Bernhard Schölkopf, Patrice Simard +2 more
It is shown that both invariances under group transformations and prior knowledge about locality in images can be incorporated by constructing appropriate kernel functions by exploring methods for incorporating prior knowledge in Support Vector learning machines.
Document Image Defect Models
285 Citations1992Henry S. Baird
Work-in-progress towards a parameterized model of local imaging defects is described, together with a variety of motivating theoretical arguments and empirical evidence, and a pseudo-random image generator implementing the model has been built.
Lecture notes in computer scienceIncorporating invariances in support vector learning machines
280 Citations1996Bernhard Schölkopf, Chris Burges +1 more
This work presents a method of incorporating prior knowledge about transformation invariances by applying transformations to support vectors, the training examples most critical for determining the classification boundary.
The MIT Press eBooksDynamic Alignment Kernels
242 Citations2000Chris Watkins
This paper shows that the scores produced by certain dynamic align ment algorithms for sequences are in fact valid kernel functions, proved by expressing the alignment scores explicitly as dot products.
Cambridge University Press eBooksApplicable Differential Geometry
230 Citations1987M. Crampin, F. A. E. Pirani
neural information processing systemsTangent Prop - A formalism for specifying selected invariances in an adaptive network
220 Citations1991Patrice Simard, Bernard Victorri +2 more
A scheme is implemented that allows a network to learn the derivative of its outputs with respect to distortion operators of their choosing, which not only reduces the learning time and the amount of training data, but also provides a powerful language for specifying what generalizations the authors wish the network to perform.
Fraunhofer-Publica (Fraunhofer-Gesellschaft)Improving the accuracy and speed of support vector learning machines
213 Citations1997Cjc Burges, Bernhard Schölkopf +3 more
International Journal of Pattern Recognition and Artificial IntelligenceBOOSTING PERFORMANCE IN NEURAL NETWORKS
184 Citations1993Harris Drucker, Robert E. Schapire +1 more
The boosting algorithm is used to construct an ensemble of neural networks that significantly improves performance (compared to a single network) in optical character recognition (OCR) problems and improved performance significantly, and, in some cases, dramatically.
Recognition and Structure from One 2D Model View: Observations on Prototypes, Object Classes and Symmetries
183 Citations1992Tomaso Poggio, Thomas Vetter
It is proved that for any bilaterally symmetric 3D object one non- accidental 2D model view is sufficient for recognition and linear transformations can be learned exactly from a small set of examples in the case of "linear object classes".
Machine LearningLearning to Recognize Volcanoes on Venus
98 Citations1998Michael C. Burl, Lars Asker +5 more
Important aspects of the application process not commonly encountered in the “toy world,” including obtaining labeled training data, the difficulties of working with pixel data, and the automatic extraction of higher-level features are discussed.
Alpha seeding for support vector machines
81 Citations2000Dennis DeCoste, Kiri L. Wagstaff
A key practical obstacle in applying support vector machines to many large-scale data mining tasks is that SVM's generally scale quadratically (or worse) in the number of examples or support vectors.
Open Research Online (The Open University)Neural-Network and k-Nearest-neighbor Classifiers
32 Citations1991Jane Bromley, Eduard Säckinger
The performance of a state-of-the-art neural network classifier for hand-written digits is compared to that of a k-nearest-neighbor method and to human performance.
Distortion-invariant recognition via jittered queries
23 Citations2002Dennis DeCoste, Michael C. Burl
This paper presents a new approach for achieving distortion-invariant recognition and classification, where instead of querying with a single pattern, a more robust query is constructed, based on the family of patterns formed by distorting the test example.
Handwritten digit recognition with a novel vision model that extracts linearly separable features
12 Citations2002Loo-Nin Teow, Kia-Fock Loe
It is shown empirically that the features extracted by the model are linearly separable over a large training set (MNIST).
The MIT Press eBooksNatural Regularization from Generative Models
9 Citations2000Nuria Oliver, Bernhard Schölkopf +1 more
Natural Regularization in SVMs
2 Citations2015Nuria Oliver, Alex Smola
A regularization theoretic analysis of the proposed Fisher kernel techniques is provided and it is shown that the Fisher kernel cor responds to a L p norm regularization.