Influence of noises added to hidden units on learning of multilayer perceptrons and structurization of networks
Systems and Computers in JapanPublished 1 January 1996Open access
Takio Kurita, Hideki Asoh, Shinji Umeyama, Shotaro Akaho, Akitaka Hosomi
Citations4
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is shown that a skeletal structure of the network emerges when independent Gaussian noises are added to inputs of hidden units during the error back-propagation learning and this network structurization was confirmed by experiments of pattern classification and logic Boolean function learning.
Abstract
Abstract This paper investigates the influence of noises added to hidden units of multilayer perceptrons. It is shown that a skeletal structure of the network emerges when independent Gaussian noises are added to inputs of hidden units during the error back‐propagation learning. By analyzing the average behavior of the backp‐ropagation learning to such noises, it is shown that the weights from hidden units to output units tend to be small and outputs of hidden units tend to be 0 or 1. This means that the network is automatically structurized by adding the noises. As the result, it is expected that the generalization ability of the network is improved. This network structurization was confirmed by experiments of pattern classification and logic Boolean function learning.
Keywords
Computer Science
NatureLearning representations by back-propagating errors
30,885 Citations1986David E. Rumelhart, Geoffrey E. Hinton +1 more
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
The MIT Press eBooksParallel Distributed Processing
15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more
Annals of EugenicsTHE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
14,727 Citations1936Ronald Aylmer Fisher
Mathematics of Control Signals and SystemsApproximation by superpositions of a sigmoidal function
13,684 Citations1989George Cybenko
Neural ComputationA Practical Bayesian Framework for Backpropagation Networks
2,934 Citations1992David Mackay
A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks that automatically embodies "Occam's razor," penalizing overflexible and overcomplex models.
Neural ComputationWhat Size Net Gives Valid Generalization?
1,555 Citations1989Eric B. Baum, David Haussler
It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution.
The Caltech Institute Archives (California Institute of Technology)Second order derivatives for network pruning: Optimal Brain Surgeon
1,489 Citations1992Babak Hassibi, David G. Stork
Of OBS, Optimal Brain Damage, and magnitude-based methods, only OBS deletes the correct weights from a trained XOR network in every case, and thus yields better generalization on test data.
Neural ComputationTraining with Noise is Equivalent to Tikhonov Regularization
1,312 Citations1995Chris Bishop
Neural ComputationFirst- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method
1,193 Citations1992Roberto Battiti
First- and second-order optimization methods for learning in feedforward neural networks are reviewed to illustrate the main characteristics of the different methods and their mutual relations.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)An empirical study of learning speed in back-propagation networks
838 Citations2018Scott E. Fahlman
A new learning algorithm is developed that is faster than standard backprop by an order of magnitude or more and that appears to scale up very well as the problem size increases.
IEEE International Conference on Neural NetworksNeural net pruning-why and how
230 Citations1988Sietsma, Dow
An approach is discussed that is based on learning with a net which is larger than the minimum size network required to solve the problem and then pruning the solution network, which gives a very useful partial answer to the question.
Neural NetworksThe optimised internal representation of multilayer classifier networks performs nonlinear discriminant analysis
121 Citations1990Andrew R. Webb, David Lowe
This paper illustrates why a nonlinear adaptive feed-forward layered network with linear output units can perform well as a pattern classification device and shows that minimising the error at the output of the network is equivalent to maximising a particular norm, the network discriminant function, at theoutput of the hidden units.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOptimized feature extraction and the Bayes decision in feed-forward classifier networks
83 Citations1991David Lowe, Andrew R. Webb
An important aspect of the approach is to exhibit how a priori information regarding nonuniform class membership, uneven distribution between train and test sets, and misclassification costs may be exploited in a regularized manner in the training phase of networks.
IEEE Transactions on Neural NetworksSynaptic weight noise during multilayer perceptron training: fault tolerance and training improvements
82 Citations1993Alan F. Murray, Peter Edwards
A mathematical model of the effects of synaptic arithmetic noise in multilayer perceptron training is developed and predictions are made regarding enhanced fault-tolerance and generalization ability and improved learning trajectory.
IEEE Transactions on Neural NetworksFunctional approximation by feed-forward networks: a least-squares approach to generalization
73 Citations1994Andrew R. Webb
This paper considers a least-squares approach to function approximation and generalization and shows that better generalization will occur if the error criterion used in training the generalizer is modified by the addition of a specific regularization term.
Estimation of generalization capability by combination of new information criterion and cross validation
31 Citations2002Yasuhiro Wada, Mitsuo Kawato
Computer simulation shows that the proposed information criterion can accurately predict the generalization capability of multilayer perceptrons, and thus the optimal number of hidden units can be determined.
Electronics LettersAnalogue noise-enhanced learning in neural network circuits
30 Citations1991Alan F. Murray
Experiments are reported which demonstrate that, whereas digital inaccuracy in neural arithmetic, in the form of bit-length limitation, degrades neural learning, analogue noise enhances it dramatically.
Neural ComputationMultilayer Perceptron Learning Optimized for On-Chip Implementation: A Noise-Robust System
28 Citations1992Alan F. Murray
Simplicity of implementation has been made the primary goal in the work described in this paper, to produce a system optimized for analog VLSI.
Nonlinear data analysis and multilayer perceptrons
28 Citations1989Hideki Asoh, Otsu
The authors extend their results to the nonlinear case and show that MLP with nonlinear elements approximates the non linear data analysis methods.
Lecture notes in computer scienceIterative weighted least squares algorithms for neural networks classifiers
22 Citations1993Takio Kurita
An approximation of nonlinear discriminant analysis by multilayer neural networks
16 Citations1990Hideki Asoh, N. Otsu
It is shown that the networks have a much stronger capability of class separation than the usual linear discriminant analysis method.
An Approximation of Nonlinear Canonical Correlation Analysis by Multilayer Perceptrons
13 Citations1994Hideki Asoh, O. TAKECHI
In ordinal CCA, linear transformations are constructed and applied to each datum from both of two information sources which make the correlation matrix CXY in the reduced dimensional spaces be diagonal and maximize J =tr(CXY).
Lecture notes in computer scienceRegularization learning of neural networks for generalization
9 Citations1993Shotaro Akaho
This paper proposes a learning method of neural networks based on the regularization method and estimates its generalization capability, which is defined as a difference between the expected risk accomplished by the learning and the truly minimum expected risk.
A structural learning by adding independent noises to hidden units
9 Citations1994Takio Kurita, Hideki Asoh +3 more
It is shown that a skeletal structure of a network emerges when independent noises are added to the inputs of the hidden units of multilayer perceptron during the learning by error backpropagation.
New Generation ComputingIterative weighted least squares algorithms for neural networks classifiers
9 Citations1994Takio Kurita
This paper discusses learning algorithms of layered neural networks from the standpoint of maximum likelihood estimation and proposes a learning algorithm which makes use of Fisher information matrix, which is non-negative, instead of Hessian matrix.