Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters

Published 1 January 1989

John S. Bridle

Citations398

TL;DR

It is shown that once the output layer of a multilayer perceptron is modified to provide mathematically correct probability distributions, and the usual squared error criterion is replaced with a probability-based score, the result is equivalent to Maximum Mutual Information training.

Abstract

One of the attractions of neural network approaches to pattern recognition is the use of a discrimination-based training method. We show that once we have modified the output layer of a multilayer perceptron to provide mathematically correct probability distributions, and replaced the usual squared error criterion with a probability-based score, the result is equivalent to Maximum Mutual Information training, which has been used successfully to improve the performance of hidden Markov models for speech recognition. If the network is specially constructed to perform the recognition computations of a given kind of stochastic model based classifier then we obtain a method for discrimination-based training of the parameters of the models. Examples include an HMM-based word discriminator, which we call an 'Alphanet'.

Keywords

Computer Science

Cognitive ScienceA Learning Algorithm for Boltzmann Machines*

3,324 Citations1985David H. Ackley, Geoffrey E. Hinton +1 more

Artificial IntelligenceConnectionist learning procedures

1,447 Citations1989Geoffrey E. Hinton

These relatively simple, gradient-descent learning procedures work well for small tasks and the new challenge is to find ways of improving their convergence rate and their generalization abilities so that they can be applied to larger, more realistic tasks.

NeurocomputingProbabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition

1,202 Citations1990John S. Bridle

Two modifications are explained: probability scoring, which is an alternative to squared error minimisation, and a normalised exponential (softmax) multi-input generalisation of the logistic non- linearity of feed-forward non-linear networks with multiple outputs.

Maximum mutual information estimation of hidden Markov model parameters for speech recognition

821 Citations2005L.R. Bahl, Peter Brown +2 more

A method for estimating the parameters of hidden Markov models of speech is described and recognition results are presented comparing this method with maximum likelihood estimation.

Neural ComputationReview of Neural Networks for Speech Recognition

480 Citations1989Richard P. Lippmann

Further work is necessary for large-vocabulary continuous-speech problems, to develop training algorithms that progressively build internal word models, and to develop compact VLSI neural net hardware.

Hidden Markov models: a guided tour

241 Citations2003A. Poritz

The main tool in hidden Markov modeling is the Baum-Welch algorithm for maximum likelihood estimation of the model parameters, which is discussed both from an intuitive point of view as an exercise in the art of counting and from a formalpoint of view via the information-theoretic Q-function.

Complex SystemsAccelerated learning in layered neural networks

211 Citations1988Sara A. Solla, Esther Levin +1 more

This work investigates the merits of using this error function over t he traditional quad ratic function for gradient descent for conrf guity problem and explains the characteristic steepness of the landscape defined by the error function in configuration space.

Neural Information Processing SystemsSupervised Learning of Probability Distributions by Neural Networks

156 Citations1987Eric B. Baum, Frank Wilczek

The back propagation algorithm for supervised learning can be generalized, put on a satisfactory conceptual footing, and very likely made more efficient by defining the values of the output and input neurons as probabilities and varying the synaptic weights in the gradient direction of the log likelihood, rather than the 'error'.

Speech CommunicationAlpha-nets: A recurrent ‘neural’ network architecture with a hidden Markov model interpretation

117 Citations1990John S. Bridle

A hidden Markov model isolated word recogniser using full likelihood scoring for each word model can be treated as a recurrent ‘neural’ network and can use back-propagation of partial derivatives to hill-climb on a measure of discriminability between words.

International Conference on Acoustics, Speech, and Signal ProcessingHow limited training data can allow a neural network to outperform an 'optimal' statistical classifier

39 Citations2003L.T. Niles, H.F. Silverman +2 more

It is concluded that there are pattern classification tasks in which an ANN is able to make better use of training data to achieve a lower error rate with a particular size training set.