Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
IEEE Transactions on Neural NetworksPublished 1 January 1994
Alan F. Murray, Peter J. Edwards
Citations198
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The effects of analog noise on the synaptic arithmetic during multilayer perceptron training is analyzed, by expanding the cost function to include noise-mediated terms, and predictions are made that fault tolerance, training quality and training trajectory should be improved by such noise-injection.
Abstract
We analyze the effects of analog noise on the synaptic arithmetic during multilayer perceptron training, by expanding the cost function to include noise-mediated terms. Predictions are made in the light of these calculations that suggest that fault tolerance, training quality and training trajectory should be improved by such noise-injection. Extensive simulation experiments on two distinct classification problems substantiate the claims. The results appear to be perfectly general for all training schemes where weights are adjusted incrementally, and have wide-ranging implications for all applications, particularly those involving "inaccurate" analog neural VLSI.
Keywords
Computer ScienceEngineering
ScienceOptimization by Simulated Annealing
44,600 Citations1983Scott Kirkpatrick, C. D. Gelatt +1 more
A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems.
Optimal Brain Damage
2,613 Citations1989Yann LeCun, John S. Denker +1 more
A class of practical and nearly optimal schemes for adapting the size of a neural network by using second-derivative information to make a tradeoff between network complexity and training set error is derived.
Parallel Networks that Learn to Pronounce English Text
1,556 Citations1989Terrence J. Sejnowski
IEEE Transactions on Neural NetworksUsing additive noise in back-propagation training
350 Citations1992Lasse Holmström, Petri Koistinen
It is not proved that the introduction of additive noise to the training vectors always improves network generalization, but the analysis suggests mathematically justified rules for choosing the characteristics of noise if additive noise is used in training.
IEEE Transactions on Systems Man and CyberneticsNoise injection into inputs in back-propagation learning
257 Citations1992Kiyotoshi Matsuoka
The author mathematically explains why and how the noise injection to inputs has such an effect and experimentally shows experimentally that the generalization capability could remarkably be enhanced by training the network with noise injected inputs.
IEEE Transactions on Neural NetworksSensitivity of feedforward neural networks to weight errors
221 Citations1990M. Stevenson, Robin Winter +1 more
An approximation is derived which expresses the probability of error for an output neuron of a large network (a network with many neurons per layer) as a function of the percentage change in the weights.
IEEE Transactions on Neural NetworksMaximally fault tolerant neural networks
126 Citations1992C. Neti, Michael Schneider +1 more
An application of neural network modeling is described for generating hypotheses about the relationships between response properties of neurons and information processing in the auditory system and a feedforward neural network model with a guaranteed level of fault tolerance is introduced.
IEEE Transactions on Neural NetworksSynaptic weight noise during multilayer perceptron training: fault tolerance and training improvements
82 Citations1993Alan F. Murray, Peter Edwards
A mathematical model of the effects of synaptic arithmetic noise in multilayer perceptron training is developed and predictions are made regarding enhanced fault-tolerance and generalization ability and improved learning trajectory.
Neural ComputationThe Effects of Precision Constraints in a Backpropagation Learning Network
77 Citations1990P.W. Hollis, John S. Harper +1 more
It is shown through simulations that a learning system of this nature can be implemented in spite of limited resolution in the analog circuits and using fixed point arithmetic to implement the backpropagation algorithm.
Journal of Theoretical BiologySpontaneous behaviour in neural networks
77 Citations1972John G. Taylor
Some considerations on the importance of spontaneous firing in animal behaviour after introduction of a parameter, called the spontaneity parameter, helpful in describing the amount of spontaneous behaviour of a neuron or net of them are concluded.
Physica D Nonlinear PhenomenaA stochastic version of the delta rule
62 Citations1990Stephen José Hanson
The delta rule is modified to include synaptic noise in the transmission of information and modification of the connection strength, and this stochastic version of the delta rule seems to promote escape from poor locally stable minima, and can improve convergence speed and likehood.
Electronics LettersAnalogue noise-enhanced learning in neural network circuits
30 Citations1991Alan F. Murray
Experiments are reported which demonstrate that, whereas digital inaccuracy in neural arithmetic, in the form of bit-length limitation, degrades neural learning, analogue noise enhances it dramatically.
Neural ComputationMultilayer Perceptron Learning Optimized for On-Chip Implementation: A Noise-Robust System
28 Citations1992Alan F. Murray
Simplicity of implementation has been made the primary goal in the work described in this paper, to produce a system optimized for analog VLSI.
Perspectives in neural computingCurvature-Driven Smoothing in Backpropagation Neural Networks
28 Citations1992C.M. Bishop
A modified error measure is proposed which can reduce the tendency to over-fit and whose properties can be controlled by a single scalar parameter.
Curvature-Driven Smoothing in Backpropagation Neural Networks
22 Citations1990Chris Bishop
Fault tolerance and redundancy of neural nets for the classification of acoustic data
8 Citations1991M.D. Emmerson, R.I. Damper +2 more
An investigation is made of the relation between the fault tolerance of a multilayer perceptron (MLP) and its redundancy as determined by the number of hidden-layer neurons (x) and results show that the single-layer perceptrons (SLP) was far less damage-resistant than any of the MLPs.
Relations between fault tolerance and internal representations for multi-layer perceptrons
5 Citations1992M.D. Emmerson, R.I. Damper
The technique of augmentation, which can increase the fault tolerance of a net of given size, is described and a number of analyses are used to understand the relations between the obtained measures of fault tolerance and the nets' internal representations.