Vicinal Risk Minimization

Published 1 January 2000

Olivier Chapelle, Jason Weston, Léon Bottou, Vladimir Vapnik

Citations221

TL;DR

It is shown how VRM provides a framework which integrates a number of existing algorithms, such as Parzen windows, Support Vector Machines, Ridge Regression, Constrained Logistic Classifiers and Tangent-Prop, and implies new algorithms for solving problems usually associated with generative models.

Abstract

The Vicinal Risk Minimization principle establishes a bridge between generative models and methods derived from the Structural Risk Minimization Principle such as Support Vector Machines or Statistical Regularization. We explain how VRM provides a framework which integrates a number of existing algorithms, such as Parzen windows, Support Vector Machines, Ridge Regression, Constrained Logistic Classifiers and Tangent-Prop. We then show how the approach implies new algorithms for solving problems usually associated with generative models. New algorithms are described for dealing with pattern recognition problems with very different pattern distributions and dealing with unlabeled data. Preliminary empirical results are presented. 1

Keywords

Computer Science

The Nature of Statistical Learning Theory

39,279 Citations1995Vladimir Vapnik

TechnometricsRidge Regression: Biased Estimation for Nonorthogonal Problems

6,562 Citations2000Arthur E. Hoerl, Robert W. Kennard

IEEE Transactions on Neural NetworksSupport vector machines for spam categorization

1,457 Citations1999Harris Drucker, Donghui Wu +1 more

The use of support vector machines in classifying e-mail as spam or nonspam is studied by comparing it to three other classification algorithms: Ripper, Rocchio, and boosting decision trees, which found SVM's performed best when using binary features.

Semi-Supervised Support Vector Machines

793 Citations1998Kristin P. Bennett, Ayhan Demiriz

A general S3VM model is proposed that minimizes both the misclassification error and the function capacity based on all the available data that can be converted to a mixed-integer program and then solved exactly using integer programming.

Neural ComputationLocal Learning Algorithms

544 Citations1992Léon Bottou, Vladimir Vapnik

A single analysis suggests that neither kNN or RBF, nor nonlocal classifiers, achieve the best compromise between locality and capacity.

TechnometricsVariable Kernel Estimates of Multivariate Densities

474 Citations1977Leo Breiman, William S. Meisel +1 more

A class of density estimates using a superposition of kernels where the kernel parameter can depend on the nearest neighbor distances is studied by the use of simulated data and their performance is superior to that of the usual Parzen estimators.

Prior Knowledge in Support Vector Kernels

293 Citations1997Bernhard Schölkopf, Patrice Simard +2 more

It is shown that both invariances under group transformations and prior knowledge about locality in images can be incorporated by constructing appropriate kernel functions by exploring methods for incorporating prior knowledge in Support Vector learning machines.

Machine LearningMaximum Entropy Discrimination

259 Citations2004Tony Jebara

A general framework for discriminative estimation based on the maximum entropy principle and its extensions is presented and preliminary experimental results are indicative of the potential in these techniques.

View PDF

neural information processing systemsTangent Prop - A formalism for specifying selected invariances in an adaptive network

220 Citations1991Patrice Simard, Bernard Victorri +2 more

A scheme is implemented that allows a network to learn the derivative of its outputs with respect to distortion operators of their choosing, which not only reduces the learning time and the amount of training data, but also provides a powerful language for specifying what generalizations the authors wish the network to perform.

Restricted Bayes Optimal Classifiers

42 Citations2000Simon Tong, Daphne Koller

This paper investigates two particular instantiations of the notion of restricted Bayes optimal classifiers, and shows that the first uses a non-parametric density estimator — Parzen Windows with Gaussian kernels — and hyperplane decision boundaries and is asymptotically equivalent to a maximal margin hyperplane classifier, a highly successful discriminative classifier.