Is Neural Machine Translation the New State of the Art?
The Prague Bulletin of Mathematical LinguisticsPublished 1 June 2017Open access
Sheila Castilho, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, Andy Way
Citations199
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Comparing the quality of NMT systems with statistical MT is compared by describing three studies using automatic and human evaluation methods by reporting increases in fluency but inconsistent results for adequacy and post-editing effort.
Abstract
Abstract This paper discusses neural machine translation (NMT), a new paradigm in the MT field, comparing the quality of NMT systems with statistical MT by describing three studies using automatic and human evaluation methods. Automatic evaluation results presented for NMT are very promising, however human evaluations show mixed results. We report increases in fluency but inconsistent results for adequacy and post-editing effort. NMT undoubtedly represents a step forward for the MT field, but one that the community should be careful not to oversell.
Keywords
Computer Science
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
Effective Approaches to Attention-based Neural Machine Translation
8,582 Citations2015Thang Luong, Hieu Pham +1 more
A global approach which always attends to all source words and a local one that only looks at a subset of source words at a time are examined, demonstrating the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.
arXiv (Cornell University)Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
5,634 Citations2016Yonghui Wu, Mike Schuster +29 more
GNMT, Google's Neural Machine Translation system, is presented, which attempts to address many of the weaknesses of conventional phrase-based translation systems and provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delicited models.
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
3,526 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
A Study of Translation Edit Rate with Targeted Human Annotation
2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more
A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
2,080 Citations2014Michael Denkowski, Alon Lavie
Meteor Universal brings language specific evaluation to previously unsupported target languages by automatically extracting linguistic resources from the bitext used to train MT systems and using a universal parameter set learned from pooling human judgments of translation quality from several language directions.
Recurrent Continuous Translation Models
1,345 Citations2013Nal Kalchbrenner, Phil Blunsom
A class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units are introduced.
chrF: character n-gram F-score for automatic MT evaluation
970 Citations2015Maja Popović
The proposed use of character n-gram F-score for automatic evaluation of machine translation output shows very promising results, especially for the CHRF3 score – for translation from English, this variant showed the highest segment-level correlations outperforming even the best metrics on the WMT14 shared evaluation task.
On Using Very Large Target Vocabulary for Neural Machine Translation
867 Citations2015Sébastien Jean, Kyunghyun Cho +2 more
It is shown that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary.
Doubly-Attentive Decoder for Multi-modal Neural Machine Translation
189 Citations2017Iacer Calixto, Qun Liu +1 more
A Multi-modal Neural Machine Translation model in which a doubly-attentive decoder naturally incorporates spatial visual features obtained using pre-trained convolutional neural networks, bridging the gap between image description and translation.
Edinburgh Neural Machine Translation Systems for WMT 16
132 Citations2016Rico Sennrich, Barry Haddow +1 more
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions
109 Citations2017Antonio Toral, Víctor M. Sánchez-Cartagena
It is found that translations produced by neural machine translation systems are considerably different, more fluent and more accurate in terms of word order compared to those produced by phrase-based systems.
Linguistic Input Features Improve Neural Machine Translation
76 Citations2016Rico Sennrich, Barry Haddow
This paper generalizes the embedding layer of the encoder in the attentional encoder--decoder architecture to support the inclusion of arbitrary features, in addition to the baseline word feature, and finds that linguistic input features improve model quality according to three metrics: perplexity, BLEU and CHRF3.
Neural versus Phrase-Based Machine Translation Quality: a Case Study
70 Citations2016Luisa Bentivogli, Arianna Bisazza +2 more
A detailed analysis of neural versus phrase-based SMT outputs is performed, leveraging high quality post-edits performed by professional translators on the IWSLT data and provides useful insights on what linguistic phenomena are best modeled by neural models.
Using Images to Improve Machine-Translating E-Commerce Product Listings.
24 Citations2017Iacer Calixto, D. L. Stein +4 more
This paper studies how a multi-modal Neural Machine Translation (NMT) model compares to two text-only approaches: a conventional state-of-the-art attentional NMT and a Statistical Machine translation (SMT) model.