Mathematical foundations of neurocomputing
Proceedings of the IEEEPublished 1 January 1990
Шун-ичи Амари
Citations229
SJR quartileQ1
SJR score6.25
SNIP11.38
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An advanced theory of learning and self-organization is proposed, covering backpropagation and its generalizations as well as the formation of topological maps and neural representations of information.
Abstract
An attempt is made to establish a mathematical theory that shows the intrinsic mechanisms, capabilities, and limitations of information processing by various architectures of neural networks. A method of statistically analyzing one-layer neural networks is given, covering the stability of associative mapping and mapping by totally random networks. A fundamental problem of statistical neurodynamics is considered in a way that is different from the spin-glass approach. A dynamic analysis of associative memory models and a general theory of neural learning, in which the learning potential function plays a role, are given. An advanced theory of learning and self-organization is proposed, covering backpropagation and its generalizations as well as the formation of topological maps and neural representations of information.>
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Proceedings of the National Academy of SciencesNeural networks and physical systems with emergent collective computational abilities.
19,349 Citations1982J. J. Hopfield
A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
The MIT Press eBooksParallel Distributed Processing
15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more
Biological CyberneticsSelf-organized formation of topologically correct feature maps
9,519 Citations1982Teuvo Kohonen
In a simple network of adaptive physical elements which receives signals from a primary event space, the signal representations are automatically mapped onto a set of output responses in such a way that the responses acquire the same topological order as that of the primary events.
Springer series in information sciencesSelf-Organization and Associative Memory
8,770 Citations1989Teuvo Kohonen
Proceedings of the National Academy of SciencesNeurons with graded response have collective computational properties like those of two-state neurons.
6,955 Citations1984J. J. Hopfield
A model for a large network of "neurons" with a graded response (or sigmoid input-output relation) is studied and collective properties in very close correspondence with the earlier stochastic model based on McCulloch - Pitts neurons are studied.
Biological Cybernetics“Neural” computation of decisions in optimization problems
6,040 Citations1985J. J. Hopfield, David W. Tank
Results of computer simulations of a network designed to solve a difficult but well-defined optimization problem-the Traveling-Salesman Problem-are presented and used to illustrate the computational power of the networks.
Neural ComputationA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
4,451 Citations1989Ronald J. Williams, David Zipser
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.
Biophysical JournalExcitatory and Inhibitory Interactions in Localized Populations of Model Neurons
3,748 Citations1972Hugh R. Wilson, Jack D. Cowan
It is proved that the existence of limit cycle dynamics in response to one class of stimuli implies theexistence of multiple stable states and hysteresis in responseTo this work, coupled nonlinear differential equations are derived for the dynamics of spatially localized populations containing both excitatory and inhibitory model neurons.
The MIT Press eBooksParallel Distributed Processing
3,227 Citations1987James L. McClelland, David E. Rumelhart +1 more
Journal of NeuroscienceTheory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex
2,950 Citations1982EL Bienenstock, L N Cooper +1 more
The development of stimulus selectivity in the primary sensory cortex of higher vertebrates is considered in a general mathematical framework and a synaptic evolution scheme of a new kind is proposed in which incoming patterns rather than converging afferents compete.
IEEE Transactions on Systems Man and CyberneticsAbsolute stability of global pattern formation and parallel memory storage by competitive neural networks
2,524 Citations1983Michael A. Cohen, Stephen Grossberg
It remains an open question whether the Lyapunov function approach, which requires a study of equilibrium points, or an alternative global approach, such as the LyAPunov functional approach, will ultimately handle all of the physically important cases.
Journal of Mathematical BiologySimplified neuron model as a principal component analyzer
2,343 Citations1982Erkki Oja
A simple linear neuron model with constrained Hebbian-type synaptic modification is analyzed and a new class of unconstrained learning rules is derived.
Biological CyberneticsDynamics of pattern formation in lateral-inhibition type neural fields
2,053 Citations1977Шун-ичи Амари
The dynamics of pattern formation is studied for lateral-inhibition type homogeneous neural fields with general connections and it is proved that there are five types of pattern dynamics.
IEEE Transactions on Electronic ComputersGeometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition
2,037 Citations1965Thomas M. Cover
It is shown that a family of surfaces having d degrees of freedom has a natural separating capacity of 2d pattern vectors, thus extending and unifying results of Winder and others on the pattern-separating capacity of hyperplanes.
Parallel Networks that Learn to Pronounce English Text
1,556 Citations1989Terrence J. Sejnowski
Neural ComputationWhat Size Net Gives Valid Generalization?
1,555 Citations1989Eric B. Baum, David Haussler
It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution.
Neural NetworksNeural networks and principal component analysis: Learning from examples without local minima
1,360 Citations1989Pierre Baldi, Kurt Hornik
The main result is a complete description of the landscape attached to E in terms of principal component analysis, showing that E has a unique minimum corresponding to the projection onto the subspace generated by the first principal vectors of a covariance matrix associated with the training patterns.
Physical review. A, General physicsSpin-glass models of neural networks
1,193 Citations1985Daniel J. Amit, Hanoch Gutfreund +1 more
Two dynamical models, proposed by Hopfield and Little to account for the collective behavior of neural networks, are analyzed and it is shown that the long-time behavior of the two models is identical, for all temperatures below a transition temperature ${T}_{c}$.
Physical Review LettersStoring Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks
1,047 Citations1985Daniel J. Amit, Hanoch Gutfreund +1 more
The Hopfield model for a neural network is studied in the limit when the number of stored patterns increases with the size of the network, as $p=\ensuremath{\alpha}N$, and it is shown that, despite its spin-glass features, the model exhibits associative memory.
Journal of Physics A Mathematical and GeneralThe space of interactions in neural network models
1,046 Citations1988E. Gardner
A local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist and is shown to be locally stable.
NatureNon-Holographic Associative Memory
1,014 Citations1969David Willshaw, O. Buneman +1 more
The features of a hologram that commend it as a model of associative memory can be improved on by other devices.
Physical Review LettersGeneralization of back-propagation to recurrent neural networks
951 Citations1987Fernando J. Pineda
An adaptive neural network with asymmetric connections is introduced that bears a resemblance to the master/slave network of Lapedes and Farber but it is architecturally simpler.
IEEE Transactions on Information TheoryThe capacity of the Hopfield associative memory
897 Citations1987Robert J. McEliece, Edward C. Posner +2 more
Techniques from coding theory are applied to study rigorously the capacity of the Hopfield associative memory, in particular to capacity under quantization of the outer-product connection matrix.
IEEE Transactions on ComputersCorrelation Matrix Memories
878 Citations1972Teuvo Kohonen
A new model for associative memory, based on a correlation matrix, is suggested, in which any part of the memorized information can be used as a key and the memories are selective with respect to accumulated data.
Proceedings of the Royal Society of London. Series B, Biological sciencesHow patterned neural connections can be set up by self-organization
834 Citations1976David Willshaw, C. von der Malsburg
Without needing to make any elaborate assumptions about its structure or about the operations its elements are to carry out, it is shown that the mappings are set up in a system- to-system rather than a cell-to-cell fashion.
Neural ComputationLearning State Space Trajectories in Recurrent Neural Networks
675 Citations1989Barak A. Pearlmutter
A procedure for finding E/wij, where E is an error functional of the temporal trajectory of the states of a continuous recurrent network and wij are the weights of that network, which seems particularly suited for temporally continuous domains.
IEEE Transactions on Electronic ComputersA Theory of Adaptive Pattern Classifiers
520 Citations1967Шун-ичи Амари
It is proved that, by the procedures proposed here, the weight vector converges to the optimal one even under nonseparable pattern distributions, and there is an important tradeoff between speed and accuracy of convergence.
Biological CyberneticsNeural theory of association and concept-formation
502 Citations1977Шун-ичи Амари
Primitive neural models of association and concept-formation are presented, which will elucidate the distributed and multiply superposed manner of retaining knowledge in the brain.
IEEE Transactions on ComputersLearning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements
460 Citations1972Шун-ичи Амари
The stability of state transition in an autonomous logical net of threshold elements is studied by the use of characteristics of thresholds elements.
Biological CyberneticsOn associative memory
445 Citations1980G Palm
The information storing capacity of certain associative and auto-associative memories is calculated and the usefulness of associative memories, as opposed to conventional listing memories, is discussed — especially in connection with brain modelling.
Neural NetworksStatistical neurodynamics of associative memory
422 Citations1988Шун-ичи Амари, Kenjiro Maginu
A new statistical neurodynamical method is proposed for analyzing the non-equilibrium dynamical behaviors of an autocorrelation associative memory model and explains the strange behaviors due to strange shapes of the basins of attractors.
Proceedings of the National Academy of SciencesFrom basic network principles to neural architecture: emergence of spatial-opponent cells.
396 Citations1986R Linsker
This paper is the first of three that address the origin and organization of feature-analyzing cells in simple systems governed by biologically plausible development rules, and introduces the theory of "modular self-adaptive networks," of which this system is an example, and explicitly demonstrates the emergence of a layer of spatial-opponent cells.
Elsevier eBooksCompetition and Cooperation in Neural Nets
379 Citations1977Шун-ичи Амари, Michael A. Arbib
The aim of this paper is to develop a mathematical study of a number of neural models as a theory of neural fields, with special attention to the analysis of equilibrium states of the system and their stability, and to the interaction of different stimuli separated in space and time.
IEEE Transactions on Systems Man and CyberneticsAssociatron-A Model of Associative Memory
366 Citations1972Kaoru Nakano
The Associatron is considered to be a simplified model of the neural network and can be constructed as a cellular structure, where each cell is connected to only its neighbor cells and all cells run in parallel.
Biological CyberneticsConvergence properties of Kohonen's topology conserving maps: fluctuations, stability, and dimension selection
325 Citations1988Helge Ritter, Klaus Schulten
A Markovian algorithm for the formation of topologically correct feature maps proposed earlier by Kohonen is analysed and an instability responsible for a phenomenon termed by Koh Finnish “automatic selection of feature dimensions” is studied.
IEEE Transactions on Systems Man and CyberneticsCharacteristics of Random Nets of Analog Neuron-Like Elements
290 Citations1972Shun-Ichi Amarimber
The dynamic behavior of randomly connected analog neuron-like elements that process pulse-frequency modulated signals is investigated from the macroscopic point of view and it is shown that a stable oscillation exists in such a net?in contrast with the fact that no stable oscillations exist in a net of statistically symmetric structure.
Advances in psychologyAbsolute Stability of Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks
279 Citations1987Michael A. Cohen, Stephen Grossberg
Discrete Applied MathematicsDecreasing energy functions as a tool for studying threshold networks
257 Citations1985Eric Goles-Chacc, Françoise Fogelman‐Soulié +1 more
Block sequential iterations of threshold networks are studied through the use of a monotonic operator, analogous to the spin glass energy, which allows to characterize the dynamics: transient and fixed points.
Neural NetworksCharacteristics of sparsely encoded associative memory
215 Citations1989Шун-ичи Амари
It is proved that the sparsely encoded associative memory has a large basin of attraction around each memorized pattern, when and only when an activity control mechanism is attached to it.
KybernetikA method of statistical neurodynamics
183 Citations1974Шун-ичи Амари
It is shown that the distance between two microstates can also be treated as a macrostate in a generalized sense and the dynamics of distance reveals interesting microscopic properties of random nerve nets, such as the stability of state-transition, the transient lengths, etc.
Physics TodayStatistical Mechanics of Neural Networks
162 Citations1988Haim Sompolinsky
A neural network is a large, highly interconnected assembly of simple elements that resemble biological neurons, which fire when the sum of the inputs from their synapses exceeds a “firing” threshold.
Biophysical JournalAssociative neural network model for the generation of temporal patterns. Theory and application to central pattern generators
157 Citations1988David Kleinfeld, Haim Sompolinsky
An associative neural network model whose dynamic behavior is similar to that of central pattern generators is described, which predicts the strength of all possible connections between pairs of neurons on the basis of the outputs of the CPG.
Journal of Theoretical BiologyBrain functions and neural dynamics
157 Citations1970E. Harth, T.J. Csermely +2 more
A model based on observations of netlets is presented, in which it is assumed that the netlets are the fundamental building blocks out of which nets of considerable complexity may be as embled and the concepts of netlet interaction are applied in conjunction with the principle of synaptic facilitation to information processing in the cortex.
Proceedings of the IEEECharacteristics of randomly connected threshold-element networks and network systems
150 Citations1971Шун-ичи Амари
The characteristics of the networks which are composed of many randomly connected threshold elements are investigated with the intention of understanding some aspects of information processing in nervous systems.
IEEE Transactions on Systems Man and CyberneticsField theory of self-organizing neural nets
149 Citations1983Шун-ичи Амари
A field theory is proposed as a mathematical method for analyzing learning and self-organizing nerve nets and systems in a unified manner and shows by the use of the theory that a nerve net has an ability for automatically forming categorizers or signal detecting cells for the signals which the net receives from its environment.
The European Physical Journal BLearning and pattern recognition in spin glass models
146 Citations1985Wolfgang Kinzel
It is demonstrated that a random network can learn patterns by reducing frustrated bonds and the Hopfield model, a simple model for neural networks, is analyzed in detail.
Biological CyberneticsFormation of topographic maps and columnar microstructures in nerve fields
142 Citations1979A. Takeuchi, Шун-ичи Амари
A mathematical analysis of a revised version of the Willshaw-Malburg model of topographic formation, solving the equations of synaptic self-organization coupled with the field equation of neural excitations proves that two cases exist depending on parameters, and that the smooth topographic organization is obtained as a stable equilibrium of the equations.
Journal of Mathematical BiologyExistence and stability of local excitations in homogeneous neural fields
135 Citations1979Kouji Kishimoto, Шун-ичи Амари
The existence of a local excitation pattern solution as well as its waveform stability is proved by the use of the Schauder fixed-point theorem and a generalized version of the Perron-Frobenius theorem of positive matrices to the function space.
Biological CyberneticsHomogeneous nets of neuron-like elements
134 Citations1975Шун-ичи Амари
Propagation and reverberation of excitation patterns are investigated for 1-dimensional and 2-dimensional homogeneousnets of neuron-like elements and an algebraic theory of general homogeneous nets is developed.
Neural NetworksAdaptive neural oscillator using continuous-time back-propagation learning
130 Citations1989Kenji Doya, Shuji Yoshizawa
A neural network model of temporal pattern memory in animal motor systems is proposed that receives an external oscillatory input with some desired wave form, then, after sufficient learning, the network autonomously oscillates in the previously given wave form.
Journal of Physics F Metal PhysicsAnalytic theory of the ground state properties of a spin glass. I. Ising spin glass
129 Citations1980Fumihiko Tanaka, S. F. Edwards
Journal of Theoretical BiologySensori-motor transformations in the brain (with a critique of the tensor theory of cerebellum)
129 Citations1985Michael A. Arbib, Шун-ичи Амари
The Pellionisz-Llinas cerebellar lookahead module is addressed, and theoretical and experimental tests of the model are suggested, and the investigation of neural analogs of the Kalman-Bucy filter is suggested to suggest fruitful new hypotheses.
Journal of Physics A Mathematical and GeneralStructure of metastable states in the Hopfield model
119 Citations1986E. Gardner
An upper bound for the number of metastable states in the Hopfield model is calculated as a function of the Hamming fraction from an input pattern, which implies that there is a gap between a set of states close to the input pattern and another set centred around the hamming fraction 0.5 from it.
SIAM Journal on Applied MathematicsA Mathematical Foundation for Statistical Neurodynamics
110 Citations1977Шун-ичи Амари, Kiyonori Yoshida +1 more
This work first elucidate the stochastic structures of random nerve nets and derives macroscopic state equations which apply to a wide range of ensembles of random nets which are shown to hold in a weak sense.
Biological CyberneticsMathematical theory on formation of category detecting nerve cells
110 Citations1978Шун-ичи Амари, Akikazu Takeuchi
The present model fits well most of the experiments on the developmental plasticity of the visual cortex such as the formation of orientation detecting cells, monocular and alternate monocular deprivation in normal and abnormal environments.
Journal de Physique LettresScaling laws for the attractors of Hopfield networks
103 Citations1985G. Weisbuch, F. Fogelman-Soulie
SIAM Journal on Applied MathematicsSome Averaging and Stability Results for Random Differential Equations
70 Citations1979Stuart Geman
Physical Review LettersExact solution of a layered neural network model
69 Citations1987Ronny Meir, Eytan Domany
Biological CyberneticsA learning algorithm to teach spatiotemporal patterns to recurrent neural networks
67 Citations1990Masa-aki Sato
A new supervised learning algorithm that teaches spatiotemporal patterns to the recurrent neural network with arbitrary feedback connections that is equivalent to the back propagation method for the recurrent network if the discrete time prescription is adopted.
KybernetikA model of associative memory in the brain
67 Citations1973Kunihiko Fukushima
A model of associative memory for time varying spatial patterns is proposed and simulated on a digital computer, and shows an ability for associatives memory similar to that of the brain.
The European Physical Journal BLearning and pattern recognition in spin glass models
56 Citations1986Wolfgang Kinzel
It is demonstrated that a random network can learn patterns by reducing frustrated bonds and the Hopfield model, a simple model for neural networks, is analyzed in detail.
Research notes in neural computingDynamical Stability of Formation of Cortical Maps
42 Citations1989Шун-ичи Амари
The mathematical theory of formation of a cortical map by self-organization is given, which explains the emergence of a microcolumnar or mosaic structure in the cerebrum and elucidates the dynamical stability of such a map.
Springer series in synergeticsAssociative Memory and Its Statistical Neurodynamical Analysis
13 Citations1988Шун-ичи Амари
Biological CyberneticsAn extended first approximation model for the amygdaloid kindling phenomenon
12 Citations1978Israel Lieblich, Шун-ичи Амари
A first approximation model, which accounts for the strongest phenomena defining kindling, is suggested, based on an excitatory-inhibitory coupling of neural aggregates, to which a self-stimulation element for theexcitatory aggregate was added.
Lecture notes in biomathematicsOn the Topological Representation of Signals in Self-Organizing Nerve Fields
1 Citations1987Шун-ичи Амари, Minoru Maruyama
The brain can self-organize its structure based on environmental information and when a set of signals is applied repeatedly to a nerve system, for each signal in the set, the system forms bySelf-organization sets of representative cells that are excited in response to particular signals but are not excited by any other signals.