Sparse Neural Networks With Large Learning Diversity
IEEE Transactions on Neural NetworksPublished 10 June 2011Open access
Vincent Gripon, Claude Berrou
Citations142
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Though the proposed network is very simple since it is based on binary neurons and binary connections, it is able to learn a large number of messages and recall them, even in presence of strong erasures.
Abstract
Coded recurrent neural networks with three levels of sparsity are introduced. The first level is related to the size of messages that are much smaller than the number of available neurons. The second one is provided by a particular coding rule, acting as a local constraint in the neural activity. The third one is a characteristic of the low final connection density of the network after the learning phase. Though the proposed network is very simple since it is based on binary neurons and binary connections, it is able to learn a large number of messages and recall them, even in presence of strong erasures. The performance of the network is assessed as a classifier and as an associative memory.
Keywords
Computer Science
Proceedings of the National Academy of SciencesNeural networks and physical systems with emergent collective computational abilities.
19,349 Citations1982J. J. Hopfield
A model of a system having a large number of simple equivalent components, based on aspects of neurobiology but readily adapted to integrated circuits, produces a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size.
SIAM ReviewThe Structure and Function of Complex Networks
18,742 Citations2003Michael Newman
Developments in this field are reviewed, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.
North-Holland mathematical libraryThe Theory of Error-Correcting Codes
11,012 Citations1977
This book presents an introduction to BCH Codes and Finite Fields, and methods for Combining Codes, and discusses self-dual Codes and Invariant Theory, as well as nonlinear Codes, Hadamard Matrices, Designs and the Golay Code.
IEEE Transactions on Information TheoryLow-density parity-check codes
10,471 Citations1962Robert G. Gallager
A simple but nonoptimum decoding scheme operating directly from the channel a posteriori probabilities is described and the probability of error using this decoder on a binary symmetric channel is shown to decrease at least exponentially with a root of the block length.
Biological CyberneticsSelf-organized formation of topologically correct feature maps
9,519 Citations1982Teuvo Kohonen
In a simple network of adaptive physical elements which receives signals from a primary event space, the signal representations are automatically mapped onto a set of output responses in such a way that the responses acquire the same topological order as that of the primary events.
Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1
6,688 Citations2002Claude Berrou, Alain Glavieux +1 more
A new class of convolutional codes called turbo-codes, whose performances in terms of bit error rate (BER) are close to the Shannon limit, is discussed.
Cognitive ScienceA Learning Algorithm for Boltzmann Machines*
3,324 Citations1985David H. Ackley, Geoffrey E. Hinton +1 more
Journal of Artificial Intelligence ResearchSolving Multiclass Learning Problems via Error-Correcting Output Codes
2,718 Citations1995Tom Dietterich, Ghulum Bakiri
It is demonstrated that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.
Current Opinion in NeurobiologySparse coding of sensory inputs
1,436 Citations2004Bruno A. Olshausen, David Field
Recent physiological recordings from sensory neurons have indicated that sparse coding could be a ubiquitous strategy employed in several different modalities across different organisms.
IEEE Transactions on Information TheoryThe capacity of the Hopfield associative memory
897 Citations1987Robert J. McEliece, Edward C. Posner +2 more
Techniques from coding theory are applied to study rigorously the capacity of the Hopfield associative memory, in particular to capacity under quantization of the outer-product connection matrix.
Lecture notes in computer scienceGood codes based on very sparse matrices
443 Citations1995David Mackay, Radford M. Neal
A new family of error-correcting codes for the binary symmetric channel is presented, designed to encode a sparse source, and are defined in terms of very sparse invertible matrices, in such a way that the decoder can treat the signal and the noise symmetrically.
Trends in NeurosciencesOrganizing principles of real-time memory encoding: neural clique assemblies and universal neural codes
164 Citations2005Longnian Lin, Remus Oşan +1 more
This work has shown that activation patterns of the neural clique assemblies can be converted to strings of binary codes that would permit universal categorizations of internal brain representations across individuals and species.
IEEE Transactions on Information TheoryNeural networks, error-correcting codes, and polynomials over the binary n-cube
129 Citations1989Jehoshua Bruck, Mario Blaum
Performing maximum-likelihood decoding in a linear block error-correcting code is shown to be equivalent to finding a global maximum of the energy function of a certain neural network, and the connection between maximization of polynomials over the n-cube is investigated.
Neural NetworksIterative retrieval of sparsely coded associative memory patterns
92 Citations1996Friedhelm Schwenker, Friedrich T. Sommer +1 more
In a finite size auto-associative memory, iterative retrieval achieves higher capacity and better error correction than one-step retrieval and is shown that in the finite model, binary Hebbian learning exhibits much better performance.
Lecture notes in computer scienceIncreasing the capacity of a hopfield network without sacrificing functionality
64 Citations1997Amos Storkey
The question addressed by this paper is, ‘Can the capacity of the Hebbian rule be increased without losing locality, incrementality or immediacy?'
Frontiers in NeuroanatomyA cortical sparse distributed coding model linking mini- and macrocolumn-scale functionality
27 Citations2010Gerard J. Rinkus
An algorithm is described for activating these codes during both learning and retrievals, which causes more similar inputs to map to more highly intersecting codes, a property which yields ultra-fast storage and retrieval.
IEEE Transactions on Neural NetworksSegmented-Memory Recurrent Neural Networks
25 Citations2009Jinmiao Chen, Narendra S. Chaudhari
This work proposes an architecture called segmented-memory recurrent neural network (SMRNN), which performs better on long-term dependency problems than conventional RNNs and theoretically analyzes how the segmented memory of SMRNN helps learning long- term temporal dependencies.
IEEE Transactions on Neural NetworksWinner-take-all neural networks using the highest threshold
19 Citations2000Ju-Ferr Yang, Chi-Ming Chen
Both theoretical analyzes and simulation results demonstrate that the practical HITNET converges faster than the existing WTA networks for a large number of competitors.
Coded Hopfield networks
14 Citations2010Claude Berrou, Vincent Gripon
Error-correcting coding is introduced in associative memories based on Hopfield networks in order to increase the learning diversity as well as the recall robustness in presence of erasures and errors.