Factorization tricks for LSTM networks

arXiv (Cornell University)Published 31 March 2017Open access

Oleksii Kuchaiev, Boris Ginsburg

DOI10.48550/arxiv.1703.10722

Citations88

TL;DR

Two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory networks are presented: "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of L STM matrix, its inputs and states into the independent groups.

Abstract

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.

Keywords

Computer Science

Deep Residual Learning for Image Recognition

222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more

This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.

View PDF

Neural ComputationLong Short-Term Memory

98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber

A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.

arXiv (Cornell University)Very Deep Convolutional Networks for Large-Scale Image Recognition

75,540 Citations2014Karen Simonyan, Andrew Zisserman

This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.

View PDF

arXiv (Cornell University)Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

5,634 Citations2016Yonghui Wu, Mike Schuster +29 more

GNMT, Google's Neural Machine Translation system, is presented, which attempts to address many of the weaknesses of conventional phrase-based translation systems and provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delicited models.

View PDF

Long short-term memory recurrent neural network architectures for large scale acoustic modeling

2,924 Citations2014Haşim Sak, Andrew Senior +1 more

The first distributed training of LSTM RNNs using asynchronous stochastic gradient descent optimization on a large cluster of machines is introduced and it is shown that a two-layer deep LSTm RNN where each L STM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance.

arXiv (Cornell University)Recurrent Neural Network Regularization

2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more

This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.

View PDF

arXiv (Cornell University)Exploring the Limits of Language Modeling

919 Citations2016Rafał Józefowicz, Oriol Vinyals +3 more

This work explores recent advances in Recurrent Neural Networks for large scale Language Modeling, and extends current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language.

View PDF

Proceedings of the National Academy of SciencesConvolutional networks for fast, energy-efficient neuromorphic computing

793 Citations2016Steven K. Esser, Paul Merolla +14 more

This approach allows the algorithmic power of deep learning to be merged with the efficiency of neuromorphic processors, bringing the promise of embedded, intelligent, brain-inspired computing one step closer.

View PDF

arXiv (Cornell University)Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

781 Citations2015Xingjian Shi, Zhourong Chen +4 more

View PDF

arXiv (Cornell University)Achieving Human Parity in Conversational Speech Recognition

477 Citations2016Wayne Xiong, Jasha Droppo +6 more

The human error rate on the widely used NIST 2000 test set is measured, and the latest automated speech recognition system has reached human parity, establishing a new state of the art, and edges past the human benchmark.

View PDF