Involving language professionals in the evaluation of machine translation
Language Resources and EvaluationPublished 14 November 2014
Maja Popović, Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Sven Schmeier, Cindy Tscherwinka
Citations19
SJR quartileQ1
SJR score0.48
SNIP1.79
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Performance of different types of translation systems has been compared and analysed via ranking, error analysis and post-editing, and results from detailed human evaluation are described.
Abstract
Significant breakthroughs in machine translation (MT) only seem possible if human translators are taken into the loop. While automatic evaluation and scoring mechanisms such as BLEU have enabled the fast development of systems, it is not clear how systems can meet real-world (quality) requirements in industrial translation scenarios today. The taraXŰ project has paved the way for wide usage of multiple MT outputs through various feedback loops in system development. The project has integrated human translators into the development process thus collecting feedback for possible improvements. This paper describes results from detailed human evaluation. Performance of different types of translation systems has been compared and analysed via ranking, error analysis and post-editing.
Keywords
Computer Science
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
MultiUN: A Multilingual Corpus from United Nation Documents
214 Citations2010Andreas Eisele, Yu Chen
The interpretation is that the 'fetal origins' hypothesis' is valid for middle-age subjects who grow up in a welfare society and was associated with increased SBP and DBP.
Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation
185 Citations2010Chris Callison-Burch, Philipp Koehn +4 more
A large-scale manual evaluation of 104 machine translation systems and 41 system combination entries was conducted, which used the ranking of these systems to measure how strongly automatic metrics correlate with human judgments of translation quality for 26 metrics.
Computational LinguisticsTowards Automatic Error Analysis of Machine Translation Output
102 Citations2011Maja Popović, Hermann Ney
A framework for automatic error analysis and classification based on the identification of actual erroneous words using the algorithms for computation of Word Error Rate and Position-independent word Error Rate is proposed, which is just a very first step towards development of automatic evaluation measures that provide more specific information of certain translation problems.
Jane: Open Source Hierarchical Translation, Extended with Reordering and Lexicon Models
79 Citations2010David Vilar, Dan J. Stein +2 more
A novel reordering model for the hierarchical phrase-based approach is introduced which further enhances translation performance, and the effect some recent extended lexicon models have on the performance of the system is analyzed.
The Prague Bulletin of Mathematical LinguisticsHjerson: An Open Source Tool for Automatic Error Classification of Machine Translation Output
73 Citations2011Maja Popović
Hjerson, a tool for automatic classification of errors in machine translation output, features the detection of five word level error classes: morphological errors, reodering errors, missing words, extra words and lexical errors.
Estimating Machine Translation Post-Editing Effort with HTER
51 Citations2010Lucia Specia, Atefeh Farzindar
Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR
51 Citations2010Chris Callison-Burch, Philipp Koehn +3 more
From Human to Automatic Error Classification for Machine Translation Output
34 Citations2011Maja Popoviand, Aljoscha Burchardt
It is shown that the proposed automatic measures correlate well with human judgments across different error classes as well as across different translation outputs on four out of five commonly used error classes.
Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations.
31 Citations2010Christian Federmann
The results show that phrase-based evaluation can be used for fast evaluation obtaining significant agreement among annotators, and illustrates that the usage of human evaluation in machine translation should be reconsidered.
Evaluate with Confidence Estimation: Machine ranking of translation outputs using grammatical features
30 Citations2011Eleftherios Avramidis, Maja Popović +2 more
An evaluation method is presented which is able to rank translation outputs with no reference translation, given only their source sentence, using several features derived from analysis of both the source and the target sentences.
Improving the post-editing experience using translation recommendation: a user study
18 Citations2010Yifan He, Yanjun Ma +3 more
Based on the performance statistics and the users’ comments, it is found that translation recommendation can reduce the workload of professional post-editors and improve the acceptance of MT in the localization industry.
Lecture notes in computer scienceMachine Translation of Legal Information and Its Evaluation
16 Citations2009Atefeh Farzindar, Guy Lapalme
The machine translation system known as TransLI (Translation of Legal Information) developed by the authors for automatic translation of Canadian Court judgments from English to French and from French to English is presented.
Studies in computational intelligenceMachine Translation at Work
4 Citations2012Aljoscha Burchardt, Cindy Tscherwinka +2 more
This chapter reports on a system that is being developed as part of taraXŰ, an ongoing joint project between industry and research partners that aims to make MT economically feasible and technically usable.