Unsupervised Statistical Machine Translation
Published 1 January 2018Open access
Mikel Artetxe, Gorka Labaka, Eneko Agirre
Citations325
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper proposes an alternative approach based on phrase-based Statistical Machine Translation (SMT) that significantly closes the gap with supervised systems, and profits from the modular architecture of SMT.
Abstract
While modern machine translation has relied on large parallel corpora, a recent line of work has managed to train Neural Machine Translation (NMT) systems from monolingual corpora only (Artetxe et al., 2018c; Despite the potential of this approach for low-resource settings, existing systems are far behind their supervised counterparts, limiting their practical interest. In this paper, we propose an alternative approach based on phrase-based Statistical Machine Translation (SMT) that significantly closes the gap with supervised systems. Our method profits from the modular architecture of SMT: we first induce a phrase table from monolingual corpora through cross-lingual embedding mappings, combine it with an n-gram language model, and fine-tune hyperparameters through an unsupervised MERT variant. In addition, iterative backtranslation improves results further, yielding, for instance, 14.08 and 26.22 BLEU points in WMT 2014 English-German and English-French, respectively, an improvement of more than 7-10 BLEU points over previous unsupervised systems, and closing the gap with supervised SMT (Moses trained on Europarl) down to 2-5 BLEU points. Our implementation is available at https:// github.com/artetxem/monoses.
Keywords
Computer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
Computational LinguisticsA Systematic Comparison of Various Statistical Alignment Models
3,928 Citations2003Franz Josef Och, Hermann Ney
An important result is that refined alignment models with a first-order dependence and a fertility model yield significantly better results than simple heuristic models.
Statistical phrase-based translation
3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more
The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
A statistical approach to machine translation
1,699 Citations1990Peter F. Brown, John Cocke +6 more
The application of the statistical approach to translation from French to English and preliminary results are described and the results are given.
arXiv (Cornell University)Exploiting Similarities among Languages for Machine Translation
1,435 Citations2013Tomáš Mikolov, Quoc V. Le +1 more
This method can translate missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data and uses distributed representation of words and learns a linear mapping between vector spaces of languages.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)A Simple, Fast, and Effective Reparameterization of IBM Model 2
839 Citations2018Chris Dyer, Victor Chahuneau +1 more
A simple log-linear reparameterization of IBM Model 2 that overcomes problems arising from Model 1’'s strong assumptions and Model 2’s overparameterization is presented.
Communities in ADDI (University of the Basque Country)A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
556 Citations2018Mikel Artetxe, Gorka Labaka +1 more
Learning bilingual word embeddings with (almost) no bilingual data
488 Citations2017Mikel Artetxe, Gorka Labaka +1 more
This work further reduces the need of bilingual resources using a very simple self-learning approach that can be combined with any dictionary-based mapping technique, and works with as little bilingual evidence as a 25 word dictionary or even an automatically generated list of numerals.
Scalable Modified Kneser-Ney Language Model Estimation
447 Citations2013Kenneth Heafield, Ivan Pouzyrevsky +2 more
An efficient algorithm to estimate large modified Kneser-Ney models including interpolation using Streaming and sorting enables the algorithm to scale to much larger models by using a fixed amount of RAM and variable amount of disk.
Adversarial Training for Unsupervised Bilingual Lexicon Induction
271 Citations2017Meng Zhang, Yang Liu +2 more
This work shows that cross-lingual connection can actually be established without any form of supervision, by formulating the problem as a natural adversarial game, and investigating techniques that are crucial to successful training.
Proceedings of the AAAI Conference on Artificial IntelligenceGeneralizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations
233 Citations2018Mikel Artetxe, Gorka Labaka +1 more
A multi-step framework of linear transformations that generalizes a substantial body of previous work is proposed that allows new insights into the behavior of existing methods, including the effectiveness of inverse regression, and design a novel variant that obtains the best published results in zero-shot bilingual lexicon extraction.
Earth Mover's Distance Minimization for Unsupervised Bilingual Lexicon Induction
144 Citations2017Meng Zhang, Yang Liu +2 more
This paper proposes to minimize their earth mover’s distance, a measure of divergence between distributions, by viewing word embedding spaces as distributions, and demonstrates the success on the unsupervised bilingual lexicon induction task.
Unsupervised Neural Machine Translation with Weight Sharing
135 Citations2018Zhen Yang, Wei Chen +2 more
This work introduces an extension by utilizing two independent encoders but sharing some partial weights which are responsible for extracting high-level representations of the input sentences, which achieves significant improvements on English-German, English-French and Chinese-to-English translation tasks.
Deciphering Foreign Language
112 Citations2011Sujith Ravi, Kevin Knight
This work frames the MT problem as a decipherment task, treating the foreign text as a cipher for English and presents novel methods for training translation models from non-parallel text.
Large Scale Decipherment for Out-of-Domain Machine Translation
55 Citations2012Qing Dou, Kevin Knight
This work applies slice sampling to Bayesian decipherment and uses a new decipherment framework to improve out-of-domain machine translation and achieves significant gains of up to 3.8 BLEU points.
Learning Translation Models from Monolingual Continuous Representations
42 Citations2015Kai Zhao, Hany Hassan +1 more
This work proposes a much faster and simpler method that directly hallucinates translation rules for infrequent phrases based on phrases with similar continuous representations for which a translation is known, and investigates approximated nearest neighbor search with redundant bit vectors which is three times faster and significantly more accurate than locality sensitive hashing.
Dependency-Based Decipherment for Resource-Limited Machine Translation
21 Citations2013Qing Dou, Kevin Knight
This work introduces dependency relations into deciphering foreign languages and shows that dependency relations help improve the state-ofthe-art deciphering accuracy by over 500%.
Unifying Bayesian Inference and Vector Space Models for Improved Decipherment
20 Citations2015Qing Dou, Ashish Vaswani +2 more
Experimental results show that the base distribution is highly beneficial to decipherment, improving state-of-the-art decipherment accuracy from 45.8% to 67.4% for Spanish/English, and from 5.1% to 11.2% for Malagasy/English.
HAL (Le Centre pour la Communication Scientifique Directe)A Bilingual Graph-Based Semantic Model for Statistical Machine Translation
16 Citations2016Rui Wang, Hai Zhao +3 more
Bilingual Graph-based Semantic Model (BGSM) is capable of effectively modeling word sense representation instead of the word form itself and can enhance SMT both in performance and efficiency in comparison against existing methods.