Regularization using jittered training data

Published 2 January 2003

Russell Reed, Sung‐Kwun Oh, Robert J. Marks

DOI10.1109/ijcnn.1992.227178

Citations59

TL;DR

The authors investigate the training of a layered perceptron with jittered data and shows that the effect of generating additional training data by adding noise to the input data introduces convolutional smoothing of the target function.

Abstract

The authors investigate the training of a layered perceptron with jittered data. They study the effect of generating additional training data by adding noise to the input data and show that is introduces convolutional smoothing of the target function. Training using such jittered data is shown, under a small variance assumption, to be equivalent to Lagrangian regularization with a derivative regularizer. Training with jitter allows regularization within the conventional layered perceptron architecture.>

Keywords

Computer Science

Proceedings of the IEEENetworks for approximation and learning

3,257 Citations1990Tomaso Poggio, Federico Girosi

Regularization theory and a theoretical framework for approximation that leads to a class of three-layer networks called regularization networks are discussed, generalized to a formulation that includes task-dependent clustering and dimensionality reduction.

Neural NetworksCreating artificial neural networks that generalize

593 Citations1991Jocelyn Sietsma, R.J.F. Dow

A technique to test the hypothesis that multilayered, feed-forward networks with few units on the first hidden layer generalize better than networks with many units in the first layer finds the hypothesis to be false for networks trained with noisy inputs.

Springer texts in electrical engineeringIntroduction to Shannon Sampling and Interpolation Theory

542 Citations1991Robert J. Marks

This study attempts to understand, generalize and extend the cardinal series of Shannon sampling theory, which addresses the fundamental issues addressed by sampling and interpolation theory.

IEEE Transactions on Neural NetworksUsing additive noise in back-propagation training

350 Citations1992Lasse Holmström, Petri Koistinen

It is not proved that the introduction of additive noise to the training vectors always improves network generalization, but the analysis suggests mathematically justified rules for choosing the characteristics of noise if additive noise is used in training.

Neural ComputationImproving the Generalization Properties of Radial Basis Function Neural Networks

328 Citations1991Chris Bishop

It is shown that a modification to the error functional allows smoothing to be introduced explicitly without significantly affecting the speed of training.

Inversion of multilayer nets

203 Citations1989Linden, Kindermann

It is concluded that recurrent nets and even time-delay nets might be invertible.

Generalization by weight-elimination applied to currency exchange rate prediction

135 Citations1991Andreas S. Weigend, David E. Rumelhart +1 more

The authors give the details of the procedure, called weight-elimination, and describe its dynamics, and clarify the meaning of the parameters involved, which was used to predict currency exchange rates.

American Mathematical MonthlyIntroduction to Vector Analysis.

107 Citations1962H. E. Chrestenson, Harry F. Davis

This book aims to meet the professional needs of the engineer or scientist, and to give the mathematician an understanding of the three-dimensional versions of the theorems of higher geometry.

IEEE Transactions on Neural NetworksFunctional approximation by feed-forward networks: a least-squares approach to generalization

73 Citations1994Andrew R. Webb

This paper considers a least-squares approach to function approximation and generalization and shows that better generalization will occur if the error criterion used in training the generalizer is modified by the addition of a specific regularization term.

Double backpropagation increasing generalization performance

56 Citations2002Harris Drucker, Y. Le Cun

It is shown that a training algorithm termed double back- Propagation improves generalization by simultaneously minimizing the normal energy term found in back-propagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients).