Correlation between Automatic Evaluation Metric Scores, Post-Editing Speed, and Some Other Factors

Published 26 August 2009

Midori Tatsumi

Citations35

Abstract

This paper summarises the results of a pilot project conducted to investigate the correlation between automatic evaluation metric scores and post-editing speed on a segment by segment basis. Firstly, the results from the comparison of various automatic metrics and post-editing speed will be reported. Secondly, further analysis is carried out by taking into consideration other relevant variables, such as text length and structures, and by means of multiple regression. It has been found that different automatic metrics achieve different levels and types of correlation with post-editing speed. We suggest that some of the source text characteristics and machine translation errors may be able to account for the gap between the automatic metric scores and post-editing speed, and may also help with understanding human post-editing process. 1

Keywords

Computer ScienceMathematics

BLEU

21,523 Citations2001Kishore Papineni, Salim Roukos +2 more

This work proposes a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run.

View PDF

A Study of Translation Edit Rate with Targeted Human Annotation

2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more

A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.

Automatic evaluation of machine translation quality using n-gram co-occurrence statistics

1,605 Citations2002George R. Doddington

NIST commissioned NIST to develop an MT evaluation facility based on the IBM work, which is now available from NIST and serves as the primary evaluation measure for TIDES MT research.

View PDF

Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes

333 Citations2001Hans P Krings

In Repairing Texts, Hans P. Krings challenges the idea that, given the effectiveness of machine translation, major costs could be reduced by using monolingual staff to post -- edit translations.

Further meta-evaluation of machine translation

269 Citations2008Chris Callison-Burch, Cameron Shaw Fordyce +3 more

This paper analyzes the translation quality of machine translation systems for 10 language pairs translating between Czech, English, French, German, Hungarian, and Spanish and uses the human judgments of the systems to analyze automatic evaluation metrics for translation quality.

View PDF

Evaluation of Machine Translation and its Evaluation

253 Citations2006Joseph Turian, Luke Shea +1 more

The unigram-based F-measure has significantly higher correlation with human judgments than recently proposed alternatives and has an intuitive graphical interpretation, which can facilitate insight into how MT systems might be improved.

Precision and recall of machine translation

227 Citations2003I. Dan Melamed, Ryan Green +1 more

Machine translation can be evaluated using precision, recall, and the F-measure, which have significantly higher correlation with human judgments than recently proposed alternatives.

View PDF

PerspectivesEYE‐TRACKING AND TRANSLATION MEMORY MATCHES

158 Citations2007Sharon O’Brien

The results demonstrate that eye‐tracking data, including pupil dilation measurements and gaze replays, in association with retrospective protocols promise to be a very effective methodology for future research into translation processes.