Montreal Neural Machine Translation Systems for WMT’15
Published 1 January 2015Open access
Sébastien Jean, Orhan Fırat, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio
Citations143
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Neural machine translation (NMT) systems have recently achieved results comparable to the state of the art on a few translation tasks, including EnglishFrench and EnglishGerman. The main purpose of the Montreal Institute for Learning Algorithms (MILA) submission to WMT'15 is to evaluate this new approach on a greater variety of language pairs. Furthermore, the human evaluation campaign may help us and the research community to better understand the behaviour of our systems. We use the RNNsearch architecture, which adds an attention mechanism to the encoderdecoder. We also leverage some of the recent developments in NMT, including the use of large vocabularies, unknown word replacement and, to a limited degree, the inclusion of monolingual language models.
Keywords
Computer Science
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
IEEE Transactions on Signal ProcessingBidirectional recurrent neural networks
10,022 Citations1997Mike Schuster, Kuldip K. Paliwal
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
arXiv (Cornell University)ADADELTA: An Adaptive Learning Rate Method
5,531 Citations2012Matthew D. Zeiler
A novel per-dimension learning rate method for gradient descent called ADADELTA that dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent is presented.
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
The mathematics of statistical machine translation: parameter estimation
4,125 Citations1993Peter F. Brown, Vincent J. Della Pietra +2 more
It is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus, given a set of pairs of sentences that are translations of one another.
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
3,778 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
Computer Speech & LanguageAn empirical study of smoothing techniques for language modeling
2,077 Citations1999Stanley F. Chen, Joshua Goodman
RWTH Publications (RWTH Aachen)Moses: Open Source Toolkit for Statistical Machine Translation
1,451 Citations2007Philipp Koehn, Hieu Hoang +12 more
Recurrent Continuous Translation Models
1,345 Citations2013Nal Kalchbrenner, Phil Blunsom
A class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units are introduced.
KenLM: Faster and Smaller Language Model Queries
1,142 Citations2011Kenneth Heafield
KenLM is a library that implements two data structures for efficient language model queries, reducing both time and memory costs and is integrated into the Moses, cdec, and Joshua translation systems.
arXiv (Cornell University)Theano: new features and speed improvements
1,035 Citations2012Frédéric Bastien, Pascal Lamblin +7 more
New features and efficiency improvements to Theano are presented, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.
On Using Very Large Target Vocabulary for Neural Machine Translation
867 Citations2015Sébastien Jean, Kyunghyun Cho +2 more
It is shown that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary.
Addressing the Rare Word Problem in Neural Machine Translation
660 Citations2015Thang Luong, Ilya Sutskever +3 more
This paper proposes and implements an effective technique to address the problem of end-to-end neural machine translation's inability to correctly translate very rare words, and is the first to surpass the best result achieved on a WMT’14 contest task.
Maxout Networks
607 Citations2013Ian Goodfellow, David Warde-Farley +3 more
A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique.
arXiv (Cornell University)On Using Monolingual Corpora in Neural Machine Translation
511 Citations2015Çaǧlar Gülçehre, Orhan Fırat +7 more
This work investigates how to leverage abundant monolingual corpora for neural machine translation to improve results for En-Fr and En-De translation and extends to high resource languages such as Cs-En and De-En.
Empirical methods for compound splitting
315 Citations2003Philipp Koehn, Kevin Knight
Methods to learn splitting rules from monolingual and parallel corpora are introduced and evaluated against a gold standard and their impact on performance of statistical MT systems is measured.
IEEE Transactions on Neural NetworksAdaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
233 Citations2008Yoshua Bengio, Jean-Sébastien Senécal
The idea is to use an adaptive n-gram model to track the conditional distributions produced by the neural network, and it is shown that a very significant speedup can be obtained on standard problems.
Aaltodoc (Aalto University)Morfessor 2.0: Python Implementation and Extensions for Morfessor Baseline
165 Citations2013Sámi Virpioja, Peter Smit +2 more
Zenodo (CERN European Organization for Nuclear Research)Audio Chord Recognition With Recurrent Neural Networks.
145 Citations2013Nicolas Boulanger-Lewandowski, Yoshua Bengio +1 more
An efficient algorithm to search for the global mode of the output distribution while taking long-term dependencies into account is devised and the resulting method is competitive with state-of-the-art approaches on the MIREX dataset in the major/minor prediction task.