Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Published 1 April 2015
Tara N. Sainath, Oriol Vinyals, Andrew Senior, Haşim Sak
Citations1,619
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper takes advantage of the complementarity of CNNs, LSTMs and DNNs by combining them into one unified architecture, and finds that the CLDNN provides a 4-6% relative improvement in WER over an LSTM, the strongest of the three individual models.
Abstract
Both Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) have shown improvements over Deep Neural Networks (DNNs) across a wide variety of speech recognition tasks. CNNs, LSTMs and DNNs are complementary in their modeling capabilities, as CNNs are good at reducing frequency variations, LSTMs are good at temporal modeling, and DNNs are appropriate for mapping features to a more separable space. In this paper, we take advantage of the complementarity of CNNs, LSTMs and DNNs by combining them into one unified architecture. We explore the proposed architecture, which we call CLDNN, on a variety of large vocabulary tasks, varying from 200 to 2,000 hours. We find that the CLDNN provides a 4-6% relative improvement in WER over an LSTM, the strongest of the three individual models.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Understanding the difficulty of training deep feedforward neural networks
12,692 Citations2010Xavier Glorot, Yoshua Bengio
The objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks, to better understand these recent relative successes and help design better algorithms in the future.
IEEE Signal Processing MagazineDeep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
10,313 Citations2012Geoffrey E. Hinton, Li Deng +9 more
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
3,778 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
2,924 Citations2014Haşim Sak, Andrew Senior +1 more
The first distributed training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machines is introduced and it is shown that a two-layer deep LSTm RNN where each L STM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance.
Large Scale Distributed Deep Networks
2,916 Citations2012Jay B. Dean, Greg S. Corrado +10 more
This paper considers the problem of training a deep network with billions of parameters using tens of thousands of CPU cores and develops two algorithms for large-scale distributed training, Downpour SGD and Sandblaster L-BFGS, which increase the scale and speed of deep network training.
Deep Neural Networks for Acoustic Modeling in Speech Recognition
1,899 Citations2012Geoffrey E. Hinton, Li Deng +9 more
This paper provides an overview of this progress and repres nts the shared views of four research groups who have had recent successes in using deep neural networks for a coustic modeling in speech recognition.
New types of deep neural network learning for speech recognition and related applications: an overview
1,194 Citations2013Li Deng, Geoffrey E. Hinton +1 more
An overview of the invited and contributed papers presented at the special session at ICASSP-2013, entitled “New Types of Deep Neural Network Learning for Speech Recognition and Related Applications,” as organized by the authors is provided.
Deep convolutional neural networks for LVCSR
1,063 Citations2013Tara N. Sainath, Abdelrahman Mohamed +2 more
This paper determines the appropriate architecture to make CNNs effective compared to DNNs for LVCSR tasks, and explores the behavior of neural network features extracted from CNNs on a variety of LVCSS tasks, comparing CNNs toDNNs and GMMs.
arXiv (Cornell University)Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
856 Citations2014Haşim Sak, Andrew Senior +1 more
Novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition are presented.
Lecture notes in computer scienceSequential Deep Learning for Human Action Recognition
776 Citations2011Moez Baccouche, Franck Mamalet +3 more
A fully automated deep model, which learns to classify human actions without using any prior knowledge is proposed, which outperforms existing deep models, and gives comparable results with the best related works.
Traffic sign recognition with multi-scale Convolutional Networks
747 Citations2011Pierre Sermanet, Yann LeCun
This work applies Convolutional Networks (ConvNets) to the task of traffic sign classification as part of the GTSRB competition, and yields the 2nd-best accuracy above the human performance.
International Conference on Learning RepresentationsHow to Construct Deep Recurrent Neural Networks
577 Citations2014Razvan Pascanu, Çaǧlar Gülçehre +2 more
Understanding how Deep Belief Networks perform acoustic modelling
270 Citations2012Abdelrahman Mohamed, Geoffrey E. Hinton +1 more
This paper illustrates how each of these three aspects contributes to the DBN's good recognition performance using both phone recognition performance on the TIMIT corpus and a dimensionally reduced visualization of the relationships between the feature vectors learned by the Dbns that preserves the similarity structure of the feature vector at multiple scales.
Ensemble deep learning for speech recognition
257 Citations2014Li Deng, John Platt
Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling
238 Citations2009Brian Kingsbury
This paper demonstrates that neural-network acoustic models can be trained with sequence classification criteria using exactly the same lattice-based methods that have been developed for Gaussian mixture HMMs, and that using a sequence classification criterion in training leads to considerably better performance.
Hierarchical Structures of Neural Networks for Phoneme Recognition
204 Citations2006Petr Schwarz, Pavel Matějka +1 more
This paper deals with phoneme recognition based on neural networks (NN), and focuses on temporal patterns (TRAPs) and novel split temporal context (STC) phoneme recognizers and investigates into tandem NN architectures.
The IBM Attila speech recognition toolkit
118 Citations2010Hagen Soltau, George Saon +1 more
The design of IBM's Attila speech recognition toolkit is described, showing how the combination of a highly modular and efficient library of low-level C++ classes with simple interfaces, an interconnection layer implemented in a modern scripting language (Python), and a standardized collection of scripts for system-building produce a flexible and scalable toolkit.
Foundations and Trends in Signal Processing: DEEP LEARNING – Methods and Applications
79 Citations2014Li Deng, Dong Yu
Joint training of convolutional and non-convolutional neural networks
76 Citations2014Hagen Soltau, George Saon +1 more
A simple modification of neural networks is described which consists in extending the commonly used linear layer structure to an arbitrary graph structure, which allows to combine the benefits of convolutional neural networks with thebenefits of regular networks.
Spatio-Temporal Convolutional Sparse Auto-Encoder for Sequence Classification
72 Citations2012Moez Baccouche, Franck Mamalet +3 more
A novel learning-based approach for video sequence classification that automatically learns a sparse shift-invariant representation of the local 2D+t salient information, without any use of prior knowledge is presented.
Asynchronous stochastic optimization for sequence training of deep neural networks
71 Citations2014Georg Heigold, Erik McDermott +3 more
This paper suggests a framework to formalize the reasoning about the asynchrony and presents experimental results on both small and large scale Voice Search tasks to validate the effectiveness and efficiency of asynchronous stochastic optimization.
IEEE Transactions on Audio Speech and Language ProcessingExemplar-Based Sparse Representation Features: From TIMIT to LVCSR
65 Citations2011Tara N. Sainath, Bhuvana Ramabhadran +3 more
This paper combines the advantages of using both small and large vocabulary tasks by taking well-established techniques used in LVCSR systems and applying them on TIMIT to establish a new baseline, creating a novel set of exemplar-based sparse representation (SR) features.
Unfolded recurrent neural networks for speech recognition
49 Citations2014George Saon, Hagen Soltau +2 more
These models are feedforward networks with the property that the unfolded layers which correspond to the recurrent layer have time-shifted inputs and tied weight matrices and can be implemented efficiently through matrix-matrix operations on GPU architectures which makes it scalable for large tasks.
Sequence classification using the high-level features extracted from deep neural networks
46 Citations2014Li Deng, Jianshu Chen
This work reports on the construction of a diverse set of DNN features, including the vectors extracted from the output layer and from various hidden layers in the DNN, and applies these features as the inputs to four types of classifiers to carry out the identical sequence classification task of phone recognition.
Semi-supervised bootstrapping approach for neural network feature extractor training
42 Citations2013Frantiseli Grezl, Martin Karafiát
With this approach, it is possible to recover 40-55% of the difference between partially and fully transcribed data (3 to 5% absolute improvement over NN trained on supervised data only) and to recover 70-85% of automatically transcribed segments with the highest confidence.
Deep scattering spectra with deep neural networks for LVCSR tasks
19 Citations2014Tara N. Sainath, Vijayaditya Peddinti +4 more
This paper explores the optimal multi-resolution time and frequency scattering operations for LVCSR tasks, and explores techniques to reduce the dimension of the DSS features, which are similar to multi- Resolution log-mel + MFCCs and similar improvements can be obtained with this representation.