The Microsoft 2017 Conversational Speech Recognition System
Published 1 April 2018Open access
Wayne Xiong, Liang‐Hong Wu, F. Alleva, Jasha Droppo, Xuedong Huang, Andreas Stolcke
Citations480
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The latest version of Microsoft's conversational speech recognition system for the Switchboard and CallHome domains is described, which adds a CNN-BLSTM acoustic model to the set of model architectures combined previously, and includes character-based and dialog session aware LSTM language models in rescoring.
Abstract
We describe the latest version of Microsoft's conversational speech recognition system for the Switchboard and CallHome domains. The system adds a CNN-BLSTM acoustic model to the set of model architectures we combined previously, and includes character-based and dialog session aware LSTM language models in rescoring. For system combination we adopt a two-stage approach, whereby acoustic model posteriors are first combined at the senone/frame level, followed by a word-level voting via confusion networks. We also added another language model rescoring step following the confusion network combination. The resulting system yields a 5.1% word error rate on the NIST 2000 Switchboard test set, and 9.8% on the CallHome subset.
Keywords
Computer Science
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
arXiv (Cornell University)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
24,381 Citations2015Sergey Ioffe, Christian Szegedy
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
Neural ComputationBackpropagation Applied to Handwritten Zip Code Recognition
11,878 Citations1989Yann LeCun, Bernhard E. Boser +5 more
This paper demonstrates how constraints from the task domain can be integrated into a backpropagation network through the architecture of the network, successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service.
Neural NetworksFramewise phoneme classification with bidirectional LSTM and other neural network architectures
5,463 Citations2005Alex Graves, Jürgen Schmidhuber
The main findings are that bidirectional networks outperform unidirectional ones, and Long Short Term Memory (LSTM) is much faster and also more accurate than both standard Recurrent Neural Nets (RNNs) and time-windowed Multilayer Perceptrons (MLPs).
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
Neural ComputationA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
4,451 Citations1989Ronald J. Williams, David Zipser
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks.
HAL (Le Centre pour la Communication Scientifique Directe)Convolutional networks for images, speech, and time series
4,359 Citations1998Yann LeCun, Yoshua Bengio
IEEE Transactions on Audio Speech and Language ProcessingFront-End Factor Analysis for Speaker Verification
3,538 Citations2010Najim Dehak, Patrick Kenny +3 more
An extension of the previous work which proposes a new speaker representation for speaker verification, a new low-dimensional speaker- and channel-dependent space is defined using a simple factor analysis, named the total variability space because it models both speaker and channel variabilities.
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
2,924 Citations2014Haşim Sak, Andrew Senior +1 more
The first distributed training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machines is introduced and it is shown that a two-layer deep LSTm RNN where each L STM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance.
Linguistic Regularities in Continuous Space Word Representations
2,882 Citations2013Tomáš Mikolov, Wen-tau Yih +1 more
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
IEEE Transactions on Acoustics Speech and Signal ProcessingPhoneme recognition using time-delay neural networks
2,625 Citations1989Alexander Waibel, Toshiyuki Hanazawa +3 more
The authors present a time-delay neural network (TDNN) approach to phoneme recognition which is characterized by two important properties: using a three-layer arrangement of simple computing units, a hierarchy can be constructed that allows for the formation of arbitrary nonlinear decision surfaces, which the TDNN learns automatically using error backpropagation.
SWITCHBOARD: telephone speech corpus for research and development
2,140 Citations1992J. Godfrey, E. Holliman +1 more
LSTM neural networks for language modeling
1,990 Citations2012Martin Sundermeyer, Ralf Schlüter +1 more
This work analyzes the Long Short-Term Memory neural network architecture on an English and a large French language modeling task and gains considerable improvements in WER on top of a state-of-the-art speech recognition system.
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
1,619 Citations2015Tara N. Sainath, Oriol Vinyals +2 more
This paper takes advantage of the complementarity of CNNs, LSTMs and DNNs by combining them into one unified architecture, and finds that the CLDNN provides a 4-6% relative improvement in WER over an LSTM, the strongest of the three individual models.
Physical Review LettersGeneralization of back-propagation to recurrent neural networks
951 Citations1987Fernando J. Pineda
An adaptive neural network with asymmetric connections is introduced that bears a resemblance to the master/slave network of Lapedes and Farber but it is architecturally simpler.
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs
861 Citations2014Frank Seide, Hao Fu +3 more
This work shows empirically that in SGD training of deep neural networks, one can, at no or nearly no loss of accuracy, quantize the gradients aggressively—to but one bit per value—if the quantization error is carried forward across minibatches (error feedback), and implements data-parallel deterministically distributed SGD by combining this finding with AdaGrad.
Speaker adaptation of neural network acoustic models using i-vectors
606 Citations2013George Saon, Hagen Soltau +2 more
This work proposes to adapt deep neural network acoustic models to a target speaker by supplying speaker identity vectors (i-vectors) as input features to the network in parallel with the regular acoustic features for ASR, comparable in performance to DNNs trained on speaker-adapted features with the advantage that only one decoding pass is needed.
Context dependent recurrent neural network language model
574 Citations2012Tomáš Mikolov, Geoffrey Zweig
This paper improves recurrent neural network language models performance by providing a contextual real-valued input vector in association with each word to convey contextual information about the sentence being modeled by performing Latent Dirichlet Allocation using a block of preceding text.
IEEE/ACM Transactions on Audio Speech and Language ProcessingFrom Feedforward to Recurrent LSTM Neural Networks for Language Modeling
503 Citations2015Martin Sundermeyer, Hermann Ney +1 more
This paper compares count models to feedforward, recurrent, and long short-term memory (LSTM) neural network variants on two large-vocabulary speech recognition tasks, and analyzes the potential improvements that can be obtained when applying advanced algorithms to the rescoring of word lattices on large-scale setups.
Studies in fuzziness and soft computingNeural Probabilistic Language Models
491 Citations2006Yoshua Bengio, Holger Schwenk +3 more
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences, and incorporates this new language model into a state-of-the-art speech recognizer of conversational speech.
arXiv (Cornell University)Achieving Human Parity in Conversational Speech Recognition
477 Citations2016Wayne Xiong, Jasha Droppo +6 more
The human error rate on the widely used NIST 2000 test set is measured, and the latest automated speech recognition system has reached human parity, establishing a new state of the art, and edges past the human benchmark.
An Introduction to Computational Networks and the Computational Network Toolkit
354 Citations2014Dong Yu, Adam Eversole +26 more
IEEE/ACM Transactions on Audio Speech and Language ProcessingVery Deep Convolutional Neural Networks for Noise Robust Speech Recognition
347 Citations2016Yanmin Qian, Mengxiao Bi +2 more
The proposed very deep CNNs can significantly reduce word error rate (WER) for noise robust speech recognition and are competitive with the long short-term memory recurrent neural networks (LSTM-RNN) acoustic model.
The microsoft 2016 conversational speech recognition system
337 Citations2017Wayne Xiong, Jasha Droppo +6 more
Microsoft's conversational speech recognition system is described, in which recent developments in neural-network-based acoustic and language modeling are combined to advance the state of the art on the Switchboard recognition task.
Elsevier eBooksPhoneme Recognition Using Time-Delay Neural Networks
301 Citations1990Alexander Waibel, Toshiyuki Hanazawa +3 more
Speech CommunicationStatistical language model adaptation: review and perspectives
300 Citations2003Jerome R. Bellegarda
An overview of the major approaches proposed to address speech recognition performance when the lexical, syntactic, or semantic characteristics of the discourse in the training and recognition tasks differ is presented.
Observations on overlap: findings and implications for automatic processing of multi-party conversation
223 Citations2001Elizabeth Shriberg, Andreas Stolcke +1 more
It is suggested that overlap is an important inherent characteristic of conversational speech that should not be ignored; on the contrary, it should be jointly modeled with acoustic and language model information in machine processing of conversation.
Very deep multilingual convolutional neural networks for LVCSR
214 Citations2016Tom Sercu, Christian Puhrsch +2 more
A very deep convolutional network architecture with up to 14 weight layers, with small 3×3 kernels, inspired by the VGG Imagenet 2014 architecture is introduced and multilingual CNNs with multiple untied layers are introduced.
THE SRI MARCH 2000 HUB-5 CONVERSATIONAL SPEECH TRANSCRIPTION SYSTEM
148 Citations2000Andreas Stolcke, Harry Bratt +7 more
SRI’s large vocabulary conversational speech r ecognition system as used in the March 2000 NIST Hub-5E evaluation is described and a generalized ROVER algorithm is applied to combine the N-best hypotheses from several systems based on different acoustic models.
Computer Speech & LanguageA recurrent error propagation network speech recognition system
141 Citations1991Tony Robinson, F. Fallside
A speaker-independent phoneme and word recognition system based on a recurrent error propagation network trained on the TIMIT database and analysis of the phoneme recognition results shows that information available from bigram and durational constraints is adequately handled within the network allowing for efficient parsing of the network output.
INSIGHTS INTO SPOKEN LANGUAGE GLEANED FROM PHONETIC TRANSCRIPTION OF THE SWITCHBOARD CORPUS
140 Citations1996Steven Greenberg, Joy Hollenback +1 more
The Switchboard Transcription Project has phonetically transcribed a portion of the Switchboard corpus in an effort to better understand the failure of phoneme-centric models for machine recognition of speech, as well as to provide a database through which to improve the performance of recognition systems focused on conversational dialogs.
Getting more mileage from web text sources for conversational speech language modeling using class-dependent mixtures
135 Citations2003Ivan Bulyko, Mari Ostendorf +1 more
This paper shows how training data can be supplemented with text from the web filtered to match the style and/or topic of the target recognition task, but also that it is possible to get bigger performance gains from the data by using class-dependent interpolation of N-grams.
Computing consensus translation from multiple machine translation systems
130 Citations2005B. Bangalore, Germán Bordel +1 more
This work evaluates five domain-independent off-the-shelf MT systems and shows that the consensus-based translation performance is equal to or better than any of the given MT systems, in terms of both objective and subjective measures.
Studies in fuzziness and soft computingNeural Probabilistic Language Models
130 Citations2006Yoshua Bengio, Holger Schwenk +3 more
IEEE Transactions on Audio Speech and Language ProcessingAdvances in speech transcription at IBM under the DARPA EARS program
125 Citations2006S.F. Chen, Brian Kingsbury +5 more
This paper describes the technical and system building advances made in IBM's speech recognition technology over the course of the Defense Advanced Research Projects Agency (DARPA) Effective Affordable Reusable Speech-to-Text (EARS) program and presents results on English conversational telephony test data from the 2003 and 2004 NIST evaluations.
Deep Convolutional Neural Networks with Layer-Wise Context Expansion and Attention
88 Citations2016Dong Yu, Wayne Xiong +5 more
A deep convolutional neural network with layer-wise context expansion and location-based attention, for large vocabulary speech recognition, and it is shown that this model outperforms both the DNN and LSTM significantly.
Elsevier eBooksConsonant Recognition by Modular Construction of Large Phonemic Time-Delay Neural Networks
59 Citations1990Alex Waibel, Hidefumi Sawai +1 more
Multi-speaker language modeling
27 Citations2004Gang Ji, Jeff Bilmes
By augmenting a normal trigram context, this work introduces an approach for conversational language modeling that considers words from other speakers when predicting words from the current one and achieves a 8.9% perplexity reduction on Switchboard and a 12.2% reduction on the ICSI Meeting Recorder data.
Linearly augmented deep neural network
21 Citations2016Pegah Ghahremani, Jasha Droppo +1 more
The resulting LA-DNN model eliminates the need for pre-training, addresses the gradient vanishing problem for deep networks, has higher capacity in modeling linear transformations, trains significantly faster than normal DNN, and produces better acoustic models.
Transcription Methods for Consistency, Volume and Efficiency
16 Citations2010Meghan Lammie Glenn, Stephanie Strassel +4 more
It is suggested that transcripts for planned speech are generally more consistent than those for spontaneous speech, and that careful transcription methods result in higher rates of agreement when compared to quick transcription methods.
Self-stabilized deep neural network
10 Citations2016Pegah Ghahremani, Jasha Droppo
It is shown that using the Self-stabilized DNN method, it no longer require initial learning rate tuning and training converges quickly with a fixed global learning rate, and speeds convergence on model parameters.