On-line learning with minimal degradation in feedforward networks
IEEE Transactions on Neural NetworksPublished 1 May 1995Open access
Vicente Ruiz de Angulo, Carme Torras
Citations33
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work explains why overtraining affects forgetting and fault tolerance, which are seen as related problems, and proposes an algorithm for solving the problem, which is called learning with minimal degradation (LMD).
Abstract
Dealing with nonstationary processes requires quick adaptation while at the same time avoiding catastrophic forgetting. A neural learning technique that satisfies these requirements, without sacrificing the benefits of distributed representations, is presented. It relies on a formalization of the problem as the minimization of the error over the previously learned input-output patterns, subject to the constraint of perfect encoding of the new pattern. Then this constrained optimization problem is transformed into an unconstrained one with hidden-unit activations as variables. This new formulation leads to an algorithm for solving the problem, which we call learning with minimal degradation (LMD). Some experimental comparisons of the performance of LMD with backpropagation are provided which, besides showing the advantages of using LMD, reveal the dependence of forgetting on the learning rate in backpropagation. We also explain why overtraining affects forgetting and fault tolerance, which are seen as related problems.
Keywords
Computer Science
Neural ComputationFast Learning in Networks of Locally-Tuned Processing Units
4,219 Citations1989John Moody, Christian J. Darken
This work proposes a network architecture which uses a single internal layer of locally-tuned processing units to learn both classification tasks and real-valued function approximations (Moody and Darken 1988).
The Psychology of learning and motivation/The psychology of learning and motivationCatastrophic Interference in Connectionist Networks: The Sequential Learning Problem
3,578 Citations1989Michael McCloskey, Neal J. Cohen
This chapter discusses the catastrophic interference in connectionist networks, and the simulation results demonstrate only that interference is catastrophic in some specific networks.
Proceedings of the IEEENetworks for approximation and learning
3,257 Citations1990Tomaso Poggio, Federico Girosi
Regularization theory and a theoretical framework for approximation that leads to a class of three-layer networks called regularization networks are discussed, generalized to a formulation that includes task-dependent clustering and dimensionality reduction.
The Psychology of Learning and Motivation
2,623 Citations2012Brian H. Ross
Optimal Brain Damage
2,613 Citations1989Yann LeCun, John S. Denker +1 more
A class of practical and nearly optimal schemes for adapting the size of a neural network by using second-derivative information to make a tradeoff between network complexity and training set error is derived.
Neural ComputationA Resource-Allocating Network for Function Interpolation
1,359 Citations1991John Platt
A network that allocates a new computational unit whenever an unusual pattern is presented to the network, which learns much faster than do those using backpropagation networks and uses a comparable number of synapses.
Psychological ReviewConnectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
877 Citations1990Roger Ratcliff
The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning.
IEEE Transactions on Neural NetworksA simple procedure for pruning back-propagation trained neural networks
664 Citations1990Ehud D. Karnin
Shadow arrays are introduced which keep track of the incremental changes to the synaptic weights during a single pass of back-propagating learning and are ordered by decreasing sensitivity numbers so that the network can be efficiently pruned by discarding the last items of the sorted list.
Neural Information Processing SystemsSkeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment
539 Citations1988Michael C. Mozer, Paul Smolensky
The basic idea is to iteratively train the network to a certain performance criterion, compute a measure of relevance that identifies which input or hidden units are most critical to performance, and automatically trim the least relevant units.
ComputerNeural nets for adaptive filtering and adaptive pattern recognition
467 Citations1988Bernard Widrow, Robin Winter
The adaptive linear combiner is described, and practical applications of the ALC in signal processing and pattern recognition are presented, and Adaptive pattern recognition using neural nets is discussed.
Improving the convergence of back-propagation learning with second-order methods
333 Citations1989Stephen Becker, Yann LeCun
It is suggested that HSP27 plays a new role in the antiapoptotic action triggered by 17β-estradiol by modulating caspase-3 activity and stabilizing ERβ in skeletal muscle cells.
The MIT Press eBooksNeural Network Design and the Complexity of Learning
211 Citations1990Joel Judd
This work formalizes a notion of learning that characterizes the training of feed-forward networks and introduces a perspective on shallow networks, called the Support Cone Interaction graph, which is helpful in distinguishing tractable from intractable subcases.
Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionist Networks
130 Citations1991Robert M. French
A simple algorithm is presented that allows a standard feedforward backpropagation network to develop semi-distributed representations, thereby significantly reducing the problem of catastrophic forgetting.
IEEE Transactions on Neural NetworksAn adaptively trained neural network
121 Citations1991Dong-Chul Park, M.A. El-Sharkawi +1 more
The resulting adaptively trained neural network (ATNN), based on nonlinear programming techniques, is shown to adapt to new training data that are in conflict with earlier training data without affecting the neural networks' response to data elsewhere.
Connectionist models and their implications: readings from cognitive science
80 Citations1988David L. Waltz, Jerome A. Feldman
This volume includes a number of articles, some of which have previously appeared in "Cognitive Science", that present a fair sample of the late-1980s research and practice in connectionist models.
Learning internal representations
67 Citations1995Jonathan Baxter
It is proved that the number of examples required to ensure good generalisation from a representation learner obeys and that gradient descent can be used to train neural network representations and experiment results are reported providing strong qualitative support for the theoretical results.
Limits to the fault-tolerance of a feedforward neural network with learning
39 Citations2002J.A.G. Nijhuis, B. Höfflinger +2 more
The results show that the effects of learning and synapse weight decay on fault tolerance largely influence the practicality of large-scale silicon implementations, and it is anticipated that some neural VLSI architectures will not be sufficiently fault tolerant.
neural information processing systemsA Cost Function for Internal Representations
33 Citations1989Anders Krogh, C. I. Thorbergsson +1 more
A cost function for learning in feed-forward neural networks is introduced which is an explicit function of the internal representation in addition to the weights which can then be formulated as two simple perceptrons and a search for internal representations.
IEEE International Conference on Neural NetworksLearning of word stress in a sub-optimal second order back-propagation neural network
31 Citations1988Ricotti, Ragazzini +1 more
The authors show an example of an efficient and easy solution, using a neural network, of a problem that cannot be easily solved with rules, of the localization of primary word stress in text-to-speech synthesis of Italian.
IEEE Transactions on Neural NetworksDesigning multilayer perceptrons from nearest-neighbor systems
21 Citations1992S.G. Smyth
The paper describes a method of producing a reasonable starting point by using a nearest-neighbor classifier and expands this method to provide a way of programming the upper layer of any network assuming the lower layers already exist.