Translation Modeling with Bidirectional Recurrent Neural Networks
Published 1 January 2014
Martin Sundermeyer, Tamer Alkhouli, Joern Wuebker, Hermann Ney
Citations182
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work presents phrase-based translation models that are more consistent with phrasebased decoding and introduces bidirectional recurrent neural models to the problem of machine translation, allowing the full source sentence to be used in the models.
Abstract
This work presents two different trans-lation models using recurrent neural net-works. The first one is a word-based ap-proach using word alignments. Second, we present phrase-based translation mod-els that are more consistent with phrase-based decoding. Moreover, we introduce bidirectional recurrent neural models to the problem of machine translation, allow-ing us to use the full source sentence in our models, which is also of theoretical inter-est. We demonstrate that our translation models are capable of improving strong baselines already including recurrent neu-ral language models on three tasks: IWSLT 2013 German→English, BOLT Arabic→English and Chinese→English. We obtain gains up to 1.6 % BLEU and 1.7 % TER by rescoring 1000-best lists. 1
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
Learning Internal Representations by Error Propagation
16,235 Citations1985David E. Rumelhart, Geoffrey E. Hinton +1 more
IEEE Transactions on Signal ProcessingBidirectional recurrent neural networks
10,022 Citations1997Mike Schuster, Kuldip K. Paliwal
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
IEEE Transactions on Neural NetworksLearning long-term dependencies with gradient descent is difficult
8,461 Citations1994Yoshua Bengio, P. Simard +1 more
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods.
Neural NetworksFramewise phoneme classification with bidirectional LSTM and other neural network architectures
5,463 Citations2005Alex Graves, Jürgen Schmidhuber
The main findings are that bidirectional networks outperform unidirectional ones, and Long Short Term Memory (LSTM) is much faster and also more accurate than both standard Recurrent Neural Nets (RNNs) and time-windowed Multilayer Perceptrons (MLPs).
Neural ComputationLearning to Forget: Continual Prediction with LSTM
5,419 Citations2000Felix A. Gers, Jürgen Schmidhuber +1 more
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
Proceedings of the IEEEBackpropagation through time: what it does and how to do it
4,904 Citations1990Paul J. Werbos
This paper first reviews basic backpropagation, a simple method which is now being widely used in areas like pattern recognition and fault diagnosis, and describes further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations or true recurrent networks, and other practical issues which arise with this method.
Statistical phrase-based translation
3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more
The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
A Study of Translation Edit Rate with Targeted Human Annotation
2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more
A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.
Computer Speech & LanguageAn empirical study of smoothing techniques for language modeling
2,077 Citations1999Stanley F. Chen, Joshua Goodman
LSTM neural networks for language modeling
1,990 Citations2012Martin Sundermeyer, Ralf Schlüter +1 more
This work analyzes the Long Short-Term Memory neural network architecture on an English and a large French language modeling task and gains considerable improvements in WER on top of a state-of-the-art speech recognition system.
Extensions of recurrent neural network language model
1,596 Citations2011Tomáš Mikolov, Stefan Kombrink +3 more
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model.
Improved backing-off for M-gram language modeling
1,490 Citations2002Reinhard Kneser, Hermann Ney
This paper proposes to use distributions which are especially optimized for the task of back-off, which are quite different from the probability distributions that are usually used for backing-off.
Recurrent Continuous Translation Models
1,345 Citations2013Nal Kalchbrenner, Phil Blunsom
A class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units are introduced.
International Conference on Artificial Intelligence and StatisticsHierarchical Probabilistic Neural Network Language Model.
840 Citations2005Fréderic Morin, Yoshua Bengio
A point-of-care POC analyzer that can rapidly measure both medications used for treatment and illicit drugs in patient saliva and shows evidence of accuracy in measuring treatment medications and relapse to drug use is shown.
Intelligent Selection of Language Model Training Data
539 Citations2010Robert C. Moore, William D. Lewis
This work addresses the problem of selecting non-domain-specific language model training data to build auxiliary language models for use in tasks such as machine translation by comparing the cross-entropy, according to domain-specific and non- domain-specifc language models, for each sentence of the text source used to produce the latter language model.
Fast and Robust Neural Network Joint Models for Statistical Machine Translation
489 Citations2014Jacob Devlin, Rabih Zbib +4 more
A novel formulation for a neural network joint model (NNJM), which augments the NNLM with a source context window, which is purely lexicalized and can be integrated into any MT decoder.
Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding
428 Citations2013Grégoire Mesnil, Xiaodong He +2 more
The results show that on this task, both types of recurrent networks outperform the CRF baseline substantially, and a bi-directional Jordantype network that takes into account both past and future dependencies among slots works best, outperforming a CRFbased baseline by 14% in relative error reduction.
Gradient-based learning algorithms for recurrent networks and their computational complexity
416 Citations1995Ronald J. Williams, David Zipser
A simple and effective hierarchical phrase reordering model
297 Citations2008Michel Galley, Christopher D. Manning
A novel hierarchical phrase reordering model aimed at improving non-local reorderings, which seamlessly integrates with a standard phrase-based system with little loss of computational efficiency is presented.
Joint Language and Translation Modeling with Recurrent Neural Networks
259 Citations2013Michael Auli, Michel Galley +2 more
This work presents a joint language and translation model based on a recurrent neural network which predicts target words based on an unbounded history of both source and target words which shows competitive accuracy compared to the traditional channel model features.
Decoding with Large-Scale Neural Language Models Improves Translation
224 Citations2013Ashish Vaswani, Yinggong Zhao +2 more
This work develops a new model that combines the neural probabilistic language model of Bengio et al., rectified linear units, and noise-contrastive estimation, and incorporates it into a machine translation system both by reranking k-best lists and by direct integration into the decoder.
Classes for fast maximum entropy training
171 Citations2002J.T. Goodman
This paper presents a speedup technique: changing the form of the model to use classes, which leads to fewer nonzero indicator functions, and faster normalization, achieving speedups of up to a factor of 35 over one of the best previous techniques.
Deep Neural Network Language Models
160 Citations2012Ebru Arısoy, Tara N. Sainath +2 more
Results on a Wall Street Journal (WSJ) task demonstrate that DNN LMs offer improvements over a single hidden layer NNLM, and are competitive with a model M language model, considered to be one of the current state-of-the-art techniques for language modeling.
Comparison of feedforward and recurrent neural network language models
141 Citations2013Martin Sundermeyer, Ilya Oparin +4 more
A simple and efficient method to normalize language model probabilities across different vocabularies is proposed, and it is shown how to speed up training of recurrent neural networks by parallelization.
Continuous space language models for statistical machine translation
127 Citations2006Holger Schwenk, Daniel Dchelotte +1 more
This work proposes to use a new statistical language model that is based on a continuous representation of the words in the vocabulary, which achieves consistent improvements in the BLEU score on the development and test data.
HAL (Le Centre pour la Communication Scientifique Directe)Continuous space translation models with neural networks
122 Citations2012Hai-Son Le, Alexandre Allauzen +1 more
Several continuous space translation models are explored, where translation probabilities are estimated using a continuous representation of translation units in lieu of standard discrete representations, jointly computed using a multi-layer neural network with a SOUL architecture.
Continuous Space Translation Models for Phrase-Based Statistical Machine Translation
114 Citations2025Holger Schwenk
Experimental evidence is provided that the approach seems to be able to infer meaningful translation probabilities for phrase pairs not seen in the training data, or even predict a list of the most likely translations given a source phrase.
Discriminative training of a neural network statistical parser
110 Citations2004James Henderson
It is shown how a parser can be trained with a discriminative learning method while still parameterizing the problem according to a generative probability model, resulting in state-of-the-art levels of performance.
Efficient lattice rescoring using recurrent neural network language models
87 Citations2014Xiaobing Liu, Yizhou Wang +3 more
Two novel lattice rescoring methods for RNNLMs are investigated, one of which uses an n-gram style clustering of history contexts and the other exploits the distance measure between hidden history vectors.
RWTH Publications (RWTH Aachen)Training Phrase Translation Models with Leaving-One-Out
85 Citations2010Joern Wuebker, Arne Mauser +1 more
A novel leaving-one-out approach to prevent over-fitting is described that allows us to train phrase models that show improved translation performance on the WMT08 Europarl German-English task.
Jane: Open Source Hierarchical Translation, Extended with Reordering and Lexicon Models
79 Citations2010David Vilar, Dan J. Stein +2 more
A novel reordering model for the hierarchical phrase-based approach is introduced which further enhances translation performance, and the effect some recent extended lexicon models have on the performance of the system is analyzed.
Improving Statistical Machine Translation with Word Class Models
41 Citations2013Joern Wuebker, Stephan Peitz +2 more
This work presents a very simple and easy to implement method for using word classes from a monolingual or bilingual training corpus to improve translation quality and shows that with word class models, the baseline can be improved by up to 1.4% BLEU and 1.0% TER.
Jane 2: Open Source Phrase-based and Hierarchical Statistical Machine Translation
40 Citations2012Joern Wuebker, Matthias Huck +6 more
Jane 2, an open source toolkit supporting both the phrase- based and the hierarchical phrase-based paradigm for statistical machine translation is presented, implemented in C++ and provides efficient decoding algorithms and data structures.
Minimum Translation Modeling with Recurrent Neural Networks
25 Citations2014Yuening Hu, Michael Auli +2 more
This work introduces recurrent neural networkbased Minimum Translation Unit (MTU) models which make predictions based on an unbounded history of previous bilingual contexts which improve the output of a phrase-based statistical machine translation system trained on WMT 2012 French-English data by up to 1.5 BLEU.
A Phrase Orientation Model for Hierarchical Machine Translation
18 Citations2013Matthias Huck, Joern Wuebker +2 more
It is described how phrase orientation probabilities can be extracted from wordaligned training data for use with hierarchical phrase inventories, and how orientations can be scored in hierarchical decoding.
Lecture notes in computer scienceText-to-text machine translation using the RECONTRA connectionist model
3 Citations1999M. Asunción Castaño, Francisco Casacuberta
Previous work exploring the capabilities of this RECONTRA model to perform text-to-text translations in limited-domain tasks are extended.