Findings of the WMT 2019 Shared Tasks on Quality Estimation
Published 1 January 2019Open access
Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, Christian Federmann
Citations94
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The WMT19 shared task on Quality Estimation is reported, the task of predicting the quality of the output of machine translation systems given just the source text and the hypothesis translations, with a novel addition is evaluating sentence-level QE against human judgments.
Abstract
We report the results of the WMT19 shared task on Quality Estimation, i.e. the task of predicting the quality of the output of machine translation systems given just the source text and the hypothesis translations. The task includes estimation at three granularity levels: word, sentence and document. A novel addition is evaluating sentence-level QE against human judgments: in other words, designing MT metrics that do not need a reference translation. This year we include three language pairs, produced solely by neural machine translation systems. Participating teams from eleven institutions submitted a variety of systems to different task variants and language pairs.
Keywords
Computer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
45,681 Citations2018Kefallinos, Dionysios, Alexandris, Georgios +6 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Biochimica et Biophysica Acta (BBA) - Protein StructureComparison of the predicted and observed secondary structure of T4 phage lysozyme
5,357 Citations1975Brian W. Matthews
Although empirical predictions based on larger numbers of known protein structure tend to be more accurate than those based on a limited sample, the improvement in accuracy is not dramatic, suggesting that the accuracy of current empirical predictive methods will not be substantially increased simply by the inclusion of more data from additional protein structure determinations.
arXiv (Cornell University)Cross-lingual Language Model Pretraining
1,616 Citations2019Guillaume Lample, Alexis Conneau
This work proposes two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingsual language model objective.
chrF: character n-gram F-score for automatic MT evaluation
970 Citations2015Maja Popović
The proposed use of character n-gram F-score for automatic evaluation of machine translation output shows very promising results, especially for the CHRF3 score – for translation from English, this variant showed the highest segment-level correlations outperforming even the best metrics on the WMT14 shared evaluation task.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)A Simple, Fast, and Effective Reparameterization of IBM Model 2
839 Citations2018Chris Dyer, Victor Chahuneau +1 more
A simple log-linear reparameterization of IBM Model 2 that overcomes problems arising from Model 1’'s strong assumptions and Model 2’s overparameterization is presented.
arXiv (Cornell University)Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
770 Citations2018Mikel Artetxe, Holger Schwenk
An architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts using a single BiLSTM encoder with a shared byte-pair encoding vocabulary for all languages, coupled with an auxiliary decoder and trained on publicly available parallel corpora.
More accurate tests for the statistical significance of result differences
379 Citations2000Alexander Yeh
It is found in a set of experiments that many commonly used tests often underestimate the significance and so are less likely to detect differences that exist between different techniques, including computationally-intensive randomization tests.
Multi-level Translation Quality Prediction with QuEst++
120 Citations2015Lucia Specia, Gustavo Henrique Paetzold +1 more
YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources
111 Citations2019Chi-kiu Lo
A significant improvement in the correlation of YiSi-1’s scores with human judgment is made by using contextual embeddings in multilingual BERT–Bidirectional Encoder Representations from Transformers to evaluate lexical semantic similarity.
Results of the WMT20 Metrics Shared Task
69 Citations2020Nitika Mathur, Johnny Tian-Zheng Wei +3 more
Transactions of the Association for Computational LinguisticsPushing the Limits of Translation Quality Estimation
67 Citations2017André F. T. Martins, Marcin Junczys-Dowmunt +4 more
A new, carefully engineered, neural model is stacked into a rich feature-based word-level quality estimation system and the output of an automatic post-editing system is used as an extra feature, obtaining striking results on WMT16.
Neural Quality Estimation of Grammatical Error Correction
63 Citations2018Shamil Chollampatt, Hwee Tou Ng
This work proposes the first neural approach to automatic quality estimation of GEC output sentences that does not employ any hand-crafted features, and shows that a state-of-the-art GEC system can be improved when quality scores are used as features for re-ranking the N-best candidates.
White Rose Research Online (University of Leeds, The University of Sheffield, University of York)deepQuest: A Framework for Neural-based Quality Estimation
36 Citations2018Julia Ive, Frédéric Blain +1 more
This work presents a neural framework that is able to accommodate neural QE approaches at these fine-grained levels and generalize them to the level of documents and applies QE models to the output of both statistical and neural MT systems for a series of European languages.
Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings
28 Citations2019Elizaveta Yankovskaya, Andre Tättar +1 more
This work combines freely available BERT and LASER multilingual embeddings to train a neural-based regression model and uses as an input features not only pre-trained embeddeds, but also log probability of any machine translation (MT) system.
Evaluation without references: IBM1 scores as evaluation metrics
26 Citations2011Maja Popović, David Vilar +2 more
A truly automatic evaluation metric based on ibm1 lexicon probabilities which does not need any reference translations is proposed, with the most promising being ibm 1 scores calculated on morphemes and pos-4grams.