Competitive learning and soft competition for vector quantizer design
IEEE Transactions on Signal ProcessingPublished 1 January 1992
E. Yair, K. Zeger, A. Gersho
Citations202
SJR quartileQ1
SJR score2.05
SNIP2.36
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
By incorporating the principles of the stochastic approach into the KLA, a deterministic VQ design algorithm, the soft competition scheme (SCS), is introduced and experimental results are presented where the SCS consistently provided better codebooks than the generalized Lloyd algorithm (GLA), even when the same computation time was used for both algorithms.
Abstract
The authors provide a convergence analysis for the Kohonen learning algorithm (KLA) with respect to vector quantizer (VQ) optimality criteria and introduce a stochastic relaxation technique which produces the global minimum but is computationally expensive. By incorporating the principles of the stochastic approach into the KLA, a deterministic VQ design algorithm, the soft competition scheme (SCS), is introduced. Experimental results are presented where the SCS consistently provided better codebooks than the generalized Lloyd algorithm (GLA), even when the same computation time was used for both algorithms. The SCS may therefore prove to be a valuable alternative to the GLA for VQ design.>
Keywords
Computer ScienceEngineering
ScienceOptimization by Simulated Annealing
44,600 Citations1983Scott Kirkpatrick, C. D. Gelatt +1 more
A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems.
The Journal of Chemical PhysicsEquation of State Calculations by Fast Computing Machines
37,074 Citations1953N. Metropolis, Arianna W. Rosenbluth +3 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceStochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
17,980 Citations1984Stuart Geman, Donald Geman
The analogy between images and statistical mechanics systems is made and the analogous operation under the posterior distribution yields the maximum a posteriori (MAP) estimate of the image given the degraded observations, creating a highly parallel ``relaxation'' algorithm for MAP estimation.
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
IEEE Transactions on Information TheoryLeast squares quantization in PCM
15,578 Citations1982Sheelagh Lloyd
The corresponding result for any finite number of quanta is derived; that is, necessary conditions are found that the quanta and associated quantization intervals of an optimum finite quantization scheme must satisfy.
The MIT Press eBooksParallel Distributed Processing
15,227 Citations1986David E. Rumelhart, James L. McClelland +1 more
Springer series in information sciencesSelf-Organization and Associative Memory
8,770 Citations1989Teuvo Kohonen
IEEE Transactions on CommunicationsAn Algorithm for Vector Quantizer Design
7,183 Citations1980Y. Linde, A. Buzo +1 more
An efficient and intuitive algorithm is presented for the design of vector quantizers based either on a known probabilistic model or on a long training sequence of data.
Detection, Estimation, And Modulation Theory
5,977 Citations1968Harry L. Van Trees
Detection, estimation, and modulation theory, Detection, estimation and modulation theorists, اطلاعات رسانی کشاورزی .
Adaptive Signal Processing
5,121 Citations1991
This volume discusses three recently developed general methodologies for designing signal predictors under nonclassical operating conditions, namely the robust predictor, the high-speed Levinson modeling, and the approximate conditional mean nonlinear predictor.
Springer series in information sciencesSelf-Organization and Associative Memory
3,820 Citations1988Teuvo Kohonen
IEEE Transactions on Acoustics Speech and Signal ProcessingPhoneme recognition using time-delay neural networks
2,625 Citations1989Alexander Waibel, Toshiyuki Hanazawa +3 more
The authors present a time-delay neural network (TDNN) approach to phoneme recognition which is characterized by two important properties: using a three-layer arrangement of simple computing units, a hierarchy can be constructed that allows for the formation of arbitrary nonlinear decision surfaces, which the TDNN learns automatically using error backpropagation.
Simulated Annealing: Theory and Applications
2,372 Citations1987Peter J. M. van Laarhoven, Emile Aarts
Performance of the simulated annealing algorithm and the relation with statistical physics and asymptotic convergence results are presented.
Biological CyberneticsAdaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors
1,916 Citations1976Stephen Grossberg
It is shown how experience can retune feature detectors to respond to a prescribed convex set of spatial patterns, and a classification of adult feature detector properties in terms of a small number of functional principles is suggested.
Signal ProcessingAdaptive signal processing
1,596 Citations1986
All of the least squares lattice algorithms discussed are presented in the form of processing architectures which use the same basic rotation cells, which not only provides a modern perspective but also allows the detailed mathematics to be reduced to a minimum whilst providing a complete specification of the associated algorithms.
KybernetikSelf-organization of orientation sensitive cells in the striate cortex
1,481 Citations1973Chr. von der Malsburg
A nerve net model for the visual cortex of higher vertebrates is presented and a simple learning procedure is shown to be sufficient for the organization of some essential functional properties of single units.
Neural NetworksAnalysis of hidden units in a layered network trained to classify sonar targets
1,111 Citations1988Rebecca Gorman, Terrence J. Sejnowski
A neural network learning procedure has been applied to the classification of sonar returns from two undersea targets, a metal cylinder and a similarly shaped rock, and network performance and classification strategy was comparable to that of trained human listeners.
Cognitive ScienceFeature Discovery by Competitive Learning*
1,073 Citations1985David E. Rumelhart, David Zipser
This paper shows how a set of feature detectors which capture important aspects of the set of stimulus input patterns are discovered and how these feature detectors form the basis of a multilayer system that serves to learn categorizations of stimulus sets which are not linearly separable.
Cognitive ScienceFeature discovery by competitive learning
890 Citations1985David E. Rumelhart, David Zipser
Biological CyberneticsCognitron: A self-organizing multilayered neural network
727 Citations1975Kunihiko Fukushima
A new hypothesis for the organization of synapses between neurons is proposed: “The synapse from neuron x to neuron y is reinforced when x fires provided that no neuron in the vicinity of y is firing stronger than y”, and a new algorithm with which a multilayered neural network is effectively organized can be deduced.
IEEE International Conference on Neural NetworksAdding a conscience to competitive learning
642 Citations1988DeSieno
The author introduces a modification of Kohonen learning that provides rapid convergence and improved representation of the input data and forms a better approximation of p(x) in many areas of pattern recognition, statistical analysis, and control.
ComputerThe 'neural' phonetic typewriter
565 Citations1988Teuvo Kohonen
A speaker-adaptive system that transcribes dictation using an unlimited vocabulary is presented that is based on a neural network processor for the recognition of phonetic units of speech.
IEEE Transactions on Information TheoryOn the structure of vector quantizers
317 Citations1982A. Gersho
Vector quantization is intrinsically superior to predictive coding, transform coding, and other suboptimal and {\em ad hoc} procedures since it achieves optimal rate distortion performance subject only to a constraint on memory or block length of the observable signal segment being encoded.
IEEE Transactions on Acoustics Speech and Signal ProcessingExperiments on neural net recognition of spoken and written text
263 Citations1988Dc Burr
Results indicate that neural networks and nearest-neighbor classifiers perform at near the same level of accuracy and a novel handwriting encoder is described.
IEEE International Conference on Neural NetworksVector quantization of images based upon the Kohonen self-organizing feature maps
239 Citations1988Nasrabadi, Feng
IEEE International Conference on Neural NetworksKohonen's self-organizing maps: exploring their computational capabilities
154 Citations1988A. Ritter, Schulten
The authors demonstrate that the computational capabilities of Kohonen's algorithm provide an unified approach to such diverse fields as sensory mappings, combinatorial optimization, and learning in motor control.
Neural Information Processing SystemsEfficient Parallel Learning Algorithms for Neural Networks
133 Citations1988Alan Kramer, Alberto Sangiovanni‐Vincentelli
Parallelizable optimization techniques such as the Polak-Ribiere method are significantly more efficient than the Backpropagation algorithm and the noisy real-valued learning problem of hand-written character recognition.
IEEE Transactions on Signal ProcessingGlobally optimal vector quantizer design by stochastic relaxation
127 Citations1992K. Zeger, J. Vaisey +1 more
A unified formulation and study of vector quantizer design methods that couple stochastic relaxation (SR) techniques with the generalized Lloyd algorithm is presented, showing that four existing techniques all fit into a general methodology for vector quantizers design aimed at finding a globally optimal solution.
IEEE Transactions on Acoustics Speech and Signal ProcessingGradient algorithms for designing predictive vector quantizers
107 Citations1986Pao‐Chi Chang, Robert M. Gray
Two gradient algorithms for designing a PVQ are developed: the steepest descent (SD) algorithm and the stochastic gradient (SG) algorithm, both of which have the property of improving the quantizer and the predictor in the sense of minimizing the distortion as measured by the average mean-squared error.
IEEE Transactions on Information TheoryMultiple local optima in vector quantizers (Corresp.)
83 Citations1982Robert M. Gray, E. Karnin
It is shown that at least three dimensions are required for a vector quantizer to outperform a scalar quantizer for this source and multiple distinct local optima are demonstrated.
IEEE Transactions on Automatic ControlConvergence of an adaptive linear estimation algorithm
56 Citations1984E. Eweda, Odile Macchi
Neural NetworksVector quantization of images based upon the kohonen self-organization feature maps
45 Citations1988Nasser M. Nasrabadi, Yushu Feng
A neural-network clustering algorithm proposed by T. Kohonen (1986, 88) is used to design a codebook for the vector quantization of images and the results are compared with coded images when the cookbook is designed by the Linde-Buzo-Gray algorithm.
International Conference on Acoustics, Speech, and Signal ProcessingSpeech dynamics and recurrent neural networks
37 Citations2003H. Bourlard, C. Wellekens
This study establishes links among connectionist models for speech recognition and compares their respective advantages, and explains relations with hidden Markov models.
Neural NetworksThe Boltzmann Perceptron Network: A soft classifier
26 Citations1990Eyal Yair, A. Gersho
The BPN is functionally equivalent to the BM in the sense that it computes class-conditional probabilities, yet it has a feedforward structure similar to a multilayer perceptron (MLP).
Adaptive Vector Quantization
17 Citations1992A. Gersho, Robert M. Gray
It is shown that the convergence theory for adaptive quantization is intimately tied to the theory of adaptive signal processing based on the stochastic gradient algorithm.
IEEE Transactions on Signal ProcessingCorrections to 'Gradient algorithms for designing predictive vector quantizers'
10 Citations1991K. Zeger
Convergence rate analysis of gradient descent algorithms for vector quantizer design is provided and an error in a paper by Chang and Gray invalidates their observation that the generalized Lloyd algorithm is a gradient descent technique but the generalized Lloyds method is a member of the related class of coordinate descent techniques.