Continuous space translation models with neural networks
HAL (Le Centre pour la Communication Scientifique Directe)Published 3 June 2012Open access
Hai-Son Le, Alexandre Allauzen, François Yvon
Citations122
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Several continuous space translation models are explored, where translation probabilities are estimated using a continuous representation of translation units in lieu of standard discrete representations, jointly computed using a multi-layer neural network with a SOUL architecture.
Abstract
International audience
Keywords
Computer Science
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
The mathematics of statistical machine translation: parameter estimation
4,125 Citations1993Peter F. Brown, Vincent J. Della Pietra +2 more
It is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus, given a set of pairs of sentences that are translations of one another.
Class-based n -gram models of natural language
2,893 Citations1992Peter F. Brown, P.V. deSouza +3 more
This work addresses the problem of predicting a word from previous words in a sample of text and discusses n-gram models based on classes of words, finding that these models are able to extract classes that have the flavor of either syntactically based groupings or semanticallybased groupings, depending on the nature of the underlying statistics.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
Extensions of recurrent neural network language model
1,596 Citations2011Tomáš Mikolov, Stefan Kombrink +3 more
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model.
Improved backing-off for M-gram language modeling
1,490 Citations2002Reinhard Kneser, Hermann Ney
This paper proposes to use distributions which are especially optimized for the task of back-off, which are quite different from the probability distributions that are usually used for backing-off.
Computational LinguisticsThe Alignment Template Approach to Statistical Machine Translation
933 Citations2004Franz Josef Och, Hermann Ney
A phrase-based statistical machine translation approach the alignment template approach is described, which allows for general many-to-many relations between words and is easier to extend than classical statistical machinetranslation systems.
A Scalable Hierarchical Distributed Language Model
852 Citations2008Andriy Mnih, Geoffrey E. Hinton
A fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data are introduced and it is shown that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models.
ArXiv.orgAn Empirical Study of Smoothing Techniques for Language Modeling
633 Citations1996Stanley F. Chen, Joshua Goodman
Edinburgh Research Explorer (University of Edinburgh)Factored Translation Models
509 Citations2007Philipp Koehn, Hieu Hoang
In a number of experiments, it is shown that factored translation models lead to better translation performance, both in terms of automatic scores, as well as more grammatical coherence.
Computer Speech & LanguageContinuous space language models
503 Citations2006Holger Schwenk
Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words and it is shown that this approach can be incorporated into a large vocabulary continuous speech recognizer using a lattice rescoring framework at a very low additional processing time.
A hierarchical Bayesian language model based on Pitman-Yor processes
498 Citations2006Yee Whye Teh
It is shown that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models.
Lecture notes in computer sciencePhrase-Based Statistical Machine Translation
365 Citations2002Richard Zens, Franz Josef Och +1 more
A translation model that is based on bilingual phrases to explicitly model the local context is presented and it is shown that this model performs better than the single-word based model.
A unigram orientation model for statistical machine translation
289 Citations2004Christoph Tillmann
The orientation model is shown to improve translation performance over two models: 1) no block re-ordering is used, and 2) the block swapping is controlled only by a language model.
Factored language models and generalized parallel backoff
283 Citations2003Jeff Bilmes, Katrin Kirchhoff
Initial perplexity results on both CallHome Arabic and on Penn Treebank Wall Street Journal articles are provided, and FLMs with GPB can produce bigrams with significantly lower perplexity, sometimes lower than highly-optimized baseline trigrams.
Computational Linguistics<i>N</i>-gram-based Machine Translation
237 Citations2006José Bernardo Mariño Acebal, Rafael E. Banchs +5 more
This article describes in detail an n-gram approach to statistical machine translation that consists of a log-linear combination of a translation model based on n- grams of bilingual units, which are referred to as tuples, along with four specific feature functions.
Computational LinguisticsMachine Translation with Inferred Stochastic Finite-State Transducers
155 Citations2004Francisco Casacuberta, Enrique Vidal
This article uses statistical alignment methods to produce a set of conventional strings from which a stochastic rational grammar (e.g., an n-gram) is inferred, which is finally converted into a finite-state transducer.
Structured Output Layer neural network language model
131 Citations2011Hai-Son Le, Ilya Oparin +3 more
A new neural network language model (NNLM) based on word clustering to structure the output vocabulary: Structured Output Layer NNLM, able to handle vocabularies of arbitrary size, hence dispensing with the design of short-lists that are commonly used in NNLMs.
Phrasetable smoothing for statistical machine translation
113 Citations2006George Foster, Roland Kühn +1 more
It is shown that any type of smoothing is a better idea than the relative-frequency estimates that are often used and the best smoothing techniques yield consistent gains of approximately 1% (absolute) according to the BLEU metric.
Machine TranslationImproving statistical MT by coupling reordering and decoding
73 Citations2007Josep Crego, José Bernardo Mariño Acebal
An elegant and efficient approach to coupling reordering and decoding in statistical machine translation, where the n-gram translation model is also employed as distortion model, showing significant improvements in translation quality with respect to monotonic search for both translation directions at a very low computational cost.
Wider context by using bilingual language models in machine translation
69 Citations2011Jan Niehues, Teresa Herrmann +2 more
The effect of bilingual language models is analyzed and where they could help to better model the translation process is shown and improvements of translation quality on German- to-English and Arabic-to-English are shown.
The Prague Bulletin of Mathematical LinguisticsNcode: an Open Source Bilingual N-gram SMT Toolkit
43 Citations2011Josep Crego, François Yvon +1 more
Results show that Ncode outperforms Moses in terms of memory requirements and translation speed, and Ncode also achieves slightly higher accuracy results.
Smooth Bilingual N-gram Translation
39 Citations2007Holger Schwenk, Marta R. Costa‐jussà +1 more
Large vocabulary SOUL neural network language models
24 Citations2011Hai Son Le, Ilya Oparin +4 more
A new training scheme is proposed for SOUL NNLMs that is based on separate training of the outof-shortlist part of the output layer, which enables using more data at each iteration of a neural network without any considerable slow-down in training and brings additional improvements in speech recognition performance.
Machine TranslationFactored bilingual n-gram language models for statistical machine translation
17 Citations2010Josep Crego, François Yvon
This work extends the n-gram-based approach to SMT by tightly integrating more general word representations, such as lemmas and morphological classes, and uses the flexible framework of FLMs to apply a number of different back-off techniques.
IWSLTLIMSI's experiments in domain adaptation for IWSLT11
16 Citations2011Thomas Lavergne, Alexandre Allauzen +2 more
This work compares several ways to adapt existing systems and resources to the TED task with mixture of language models and tries to provide an analysis of the modest gains obtained by training a log linear combination of in- and out-of-domain models.
Morphological and syntactic features for Arabic speech recognition
14 Citations2010Hong-Kwang Jeff Kuo, Lidia Mangu +2 more
Using morphological and syntactic features, including part-of-speech tags, shallow parse tags, and exposed head words and their non-terminal labels both before and after the word to be predicted, can improve the word error rate significantly on various test sets.
Improving LVCSR system combination using neural network language model cross adaptation
11 Citations2011Xiaobing Liu, Mark Gales +1 more
Previous research on multi-level n-gram LM cross adaptation is extended to further include the cross adaptation of neural network LMs in this paper, and significant error rate gains were obtained when combining a range of Chinese LVCSR sub-systems used in the 2010 and 2011 DARPA GALE evaluations.
LIMSI @ WMT11
9 Citations2011Alexandre Allauzen, Hélène Bonneau-Maynard +9 more
This paper describes LIMSI's submissions to the Sixth Workshop on Statistical Machine Translation, and investigates the use of the SOUL language model in Machine Translation and shows significant improvements with a 10-gram SOUL model.