Making Deep Belief Networks effective for large vocabulary continuous speech recognition
Published 1 December 2011
Tara N. Sainath, Brian Kingsbury, Bhuvana Ramabhadran, Petr Fousek, Petr Novák, Abdelrahman Mohamed
Citations191
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper explores the performance of DBNs in a state-of-the-art LVCSR system, showing improvements over Multi-Layer Perceptrons (MLPs) and GMM/HMMs across a variety of features on an English Broadcast News task.
Abstract
To date, there has been limited work in applying Deep Belief Networks (DBNs) for acoustic modeling in LVCSR tasks, with past work using standard speech features. However, a typical LVCSR system makes use of both feature and model-space speaker adaptation and discriminative training. This paper explores the performance of DBNs in a state-of-the-art LVCSR system, showing improvements over Multi-Layer Perceptrons (MLPs) and GMM/HMMs across a variety of features on an English Broadcast News task. In addition, we provide a recipe for data parallelization of DBN training, showing that data parallelization can provide linear speed-up in the number of machines, without impacting WER.
Keywords
Computer Science
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
IEEE Transactions on Audio Speech and Language ProcessingContext-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition
3,052 Citations2011George E. Dahl, Dong Yu +2 more
A pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output that can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs.
Lecture notes in computer scienceA Practical Guide to Training Restricted Boltzmann Machines
2,802 Citations2012Geoffrey E. Hinton
This guide is an attempt to share expertise at training restricted Boltzmann machines with other machine learning researchers.
Kluwer Academic Publishers eBooksConnectionist Speech Recognition: A Hybrid Approach
1,136 Citations1993Hervé Bourlard, Nelson Morgan
Parallelized Stochastic Gradient Descent
1,074 Citations2010Martin Zinkevich, Markus Weimer +2 more
This paper presents the first parallel stochastic gradient descent algorithm including a detailed analysis and experimental evidence and introduces a novel proof technique — contractive mappings to quantify the speed of convergence of parameter distributions to their asymptotic limits.
Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine
275 Citations2010George E. Dahl, Marc’Aurelio Ranzato +2 more
This work uses the mean-covariance restricted Boltzmann machine (mcRBM) to learn features of speech data that serve as input into a standard DBN, and achieves a phone error rate superior to all published results on speaker-independent TIMIT to date.
Distributed Training Strategies for the Structured Perceptron
265 Citations2010Ryan McDonald, Keith Hall +1 more
This paper investigates distributed training strategies for the structured perceptron as a means to reduce training times when computing clusters are available and looks at two strategies and provides convergence bounds for a particular mode of distributed structured perceptrons training based on iterative parameter mixing (or averaging).
Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling
238 Citations2009Brian Kingsbury
This paper demonstrates that neural-network acoustic models can be trained with sequence classification criteria using exactly the same lattice-based methods that have been developed for Gaussian mixture HMMs, and that using a sequence classification criterion in training leads to considerably better performance.
Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal ProcessingOptimizing bottle-neck features for lvcsr
131 Citations2008František Grézl, Petr Fousek
Improvements over standard cepstral features and probabilistic MLP features are shown for different tasks and different neural net input representations, and a significant improvement is observed when phoneme MLP training targets are replaced by phoneme states and when delta features are added.
Conversational speech transcription using context-dependent deep neural networks
117 Citations2012Dong Yu, Frank Seide +1 more
Tandem acoustic modeling in large-vocabulary recognition
78 Citations2002Daniel P. W. Ellis, Rita Singh +1 more
It is found that, when context-independent models are used, the tandem features continue to result in large reductions in word-error rates relative to those achieved by systems using standard MFC or PLP features, but these improvements do not carry over to context-dependent models.
An architecture for rapid decoding of large vocabulary conversational speech
40 Citations2003George Saon, Geoffrey Zweig +3 more
The architecture proposed is based on classical HMM Viterbi decoding, but uses an extremely fast initial speaker-independent decoding to estimate VTL warp factors, feature-space and model-space MLLR transformations that are used in a final speaker-adapted decoding.
Parallel training of neural networks for speech recognition
28 Citations2010Karel Veselý, Lukáš Burget +1 more
A new parallel-training tool TNet was designed and optimized for multiprocessor computers and the training acceleration rates are reported on a phoneme-state classification task.