Results of the WMT14 Metrics Shared Task
Published 1 January 2014Open access
Matouš Macháċek, Ondřej Bojar
Citations109
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper presents the results of the WMT14 Metrics Shared Task, which asked participants of this task to score the outputs of the MT systems involved in W MT14 Shared Translation Task to evaluate system level correlation and segment level correlation.
Abstract
This paper presents the results of the WMT14 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in WMT14 Shared Translation Task. We collected scores of 23 metrics from 12 research groups. In addition to that we computed scores of 6 standard metrics (BLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric's scores correlate with WMT14 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence). DISCOTK-PARTY .970 .010 .921 .024 .862 .
Keywords
Computer Science
A Study of Translation Edit Rate with Targeted Human Annotation
2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more
A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
2,080 Citations2014Michael Denkowski, Alon Lavie
Meteor Universal brings language specific evaluation to previously unsupported target languages by automatically extracting linguistic resources from the bitext used to train MT systems and using a universal parameter set learned from pooling human judgments of translation quality from several language directions.
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
1,605 Citations2002George R. Doddington
NIST commissioned NIST to develop an MT evaluation facility based on the IBM work, which is now available from NIST and serves as the primary evaluation measure for TIDES MT research.
Findings of the 2014 Workshop on Statistical Machine Translation
527 Citations2014Ondřej Bojar, Christian Buck +11 more
The results of the WMT14 shared tasks, which included a standard news translation task, a separate medical translationtask, a task for run-time estimation of machine translation quality, and a metrics task, are presented.
Findings of the 2009 workshop on statistical machine translation
351 Citations2009Chris Callison-Burch, Philipp Koehn +2 more
A large-scale manual evaluation of 103 machine translation systems submitted by 34 teams was conducted, which used the ranking of these systems to measure how strongly automatic metrics correlate with human judgments of translation quality for 12 evaluation metrics.
Findings of the 2015 Workshop on Statistical Machine Translation
282 Citations2015Ondřej Bojar, Rajen Chatterjee +12 more
This paper presents the results of the WMT15 shared tasks, which included a standard news translation task, a metricstask, a tuning task,A task for run-time estimation of machine translation quality, and an automatic post-editing task.
Manual and automatic evaluation of machine translation between European languages
276 Citations2006Philipp Koehn, Christof Monz
This work evaluated machine translation performance for six European language pairs that participated in a shared task: translating French, German, Spanish texts to English and back.
A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU
184 Citations2014Boxing Chen, Colin Cherry
Three of them are first proposed in this paper and they correlate better with human judgments on the sentence-level than other smoothing techniques, and the performance of using them in statistical machine transla- tion tuning is compared.
Conference of the European Chapter of the Association for Computational LinguisticsCDER: Efficient MT Evaluation Using Block Movements.
111 Citations2006Gregor Leusch, Nicola Ueffing +1 more
Assessment of the bacteriology and antibiotic susceptibility of breast implant-related infections at two tertiary care hospitals in the Texas Medical Center found that choosing an antibiotic with anti-methicillin-resistant S. aureus activity is justified for empiric treatment of Breast implant infections, until culture and sensitivity data, if obtained, become available.
Results of the WMT14 Metrics Shared Task
109 Citations2014Matouš Macháċek, Ondřej Bojar
This paper presents the results of the WMT14 Metrics Shared Task, which asked participants of this task to score the outputs of the MT systems involved in W MT14 Shared Translation Task to evaluate system level correlation and segment level correlation.
Workshop on Statistical Machine TranslationResults of the WMT13 Metrics Shared Task
57 Citations2013Matouš Macháċek, Ondřej Bojar
DiscoTK: Using Discourse Structure for Machine Translation Evaluation
39 Citations2014Shafiq Joty, Francisco Guzmán +2 more
Novel automatic metrics for machine translation evaluation that use discourse structure and convolution kernels to compare the discourse tree of an automatic translation with that of the human reference are presented.
Enlighten: Publications (The University of Glasgow)The Reliability of the ITU-P.85 Standard for the Evaluation of Text-to-Speech Systems
27 Citations2002Yolanda Vazquez-Alvarez, Mark Huckvale
IPA and STOUT: Leveraging Linguistic and Source-based Features for Machine Translation Evaluation
19 Citations2014Meritxell Gonzàlez, Alberto Barrón‐Cedeño +1 more
The two UPC submissions to the WMT14 Metrics Shared Task take advantage of novel metrics that consider linguistic structures, lexical relationships, and semantics to compare both source and reference translation against the candidate translation.
VERTa participation in the WMT14 Metrics Task
11 Citations2014Elisabet Comelles, Jordi Atserias
VERTa, a linguistically-motivated metric that combines linguistic features at different levels, is presented and the linguistic motivation on which the metric is based is provided, as well as the different modules in VERTa and how they are combined.
Tolerant BLEU: a Submission to the WMT14 Metrics Task
5 Citations2014Jindřich Libovický, Pavel Pecina
The proposed tBLEU metric is designed to be more tolerant to errors in inflection, and therefore be more suitable for measuring quality of translation into morphologically richer languages.
Application of Prize based on Sentence Length in Chunk-based Automatic Evaluation of Machine Translation
2 Citations2014Hiroshi Echizen‐ya, Kenji Araki +1 more
This paper proposes a new automatic evaluation metric for machine translation based on chunking between the reference and candidate translation that shows stable correlation with human judgment and applies a prize based on sentence-length to the metric.
Lecture notes in computer scienceELEXR: Automatic Evaluation of Machine Translation Using Lexical Relationships
1 Citations2013Alireza Mahmoudi, Heshaam Faili +2 more
ELEXR can evaluate MT by using only one Ref assigned to each Cand without incorporating any semantic annotated resources like WordNet, and outperforms baselines, TER and BLEU, on average at system-level correlation with human judgments.