Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
FigsharePublished 29 June 2018Open access
Jonathan H. Clark, Chris Dyer, Alon Lavie, Noah A. Smith
Citations462
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper provides a systematic analysis of the effects of optimizer instability---an extraneous variable that is seldom controlled for---on experimental outcomes, and makes recommendations for reporting results more accurately.
Abstract
In statistical machine translation, a researcher seeks to determine whether some innovation (e.g., a new feature, model, or inference algorithm) improves translation quality in comparison to a baseline system. To answer this question, he runs an experiment to evaluate the behavior of the two systems on held-out data. In this paper, we consider how to make such experiments more statistically reliable. We provide a systematic analysis of the effects of optimizer instability—an extraneous variable that is seldom controlled for—on experimental outcomes, and make recommendations for reporting results more accurately
Keywords
Computer Science
The Annals of StatisticsBootstrap Methods: Another Look at the Jackknife
17,446 Citations1979B. Efron
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments
3,723 Citations2005Satanjeev Banerjee, Alon Lavie
METEOR is described, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machineproduced translation and human-produced reference translations and can be easily extended to include more advanced matching strategies.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
A Study of Translation Edit Rate with Targeted Human Annotation
2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more
A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.
Statistical Significance Tests for Machine Translation Evaluation.
1,465 Citations2004Philipp Koehn
Online Passive-Aggressive Algorithms
1,449 Citations2006Koby Crammer, Ofer Dekel +3 more
Computational LinguisticsHierarchical Phrase-Based Translation
1,164 Citations2007David Chiang
A statistical machine translation model that uses hierarchical phrasesphrases that contain subphrasing that is formally a synchronous context-free grammar but is learned from a parallel text without any syntactic annotations is presented.
All of Statistics: A Concise Course in Statistical Inference
1,035 Citations2012Larry Wasserman
This book covers a much wider range of topics than a typical introductory text on mathematical statistics, and includes modern topics like nonparametric curve estimation, bootstrapping and classification, topics that are usually relegated to follow-up courses.
Journal of the Operational Research SocietyComputer Intensive Methods for Testing Hypotheses: An Introduction
667 Citations1990K. J. Evans, Eric W. Noreen
Online large-margin training of syntactic and structural translation features
242 Citations2008David Chiang, Yuval Marton +1 more
This work explores the use of the MIRA algorithm of Crammer et al. as an alternative to MERT and shows that by parallel processing and exploiting more of the parse forest, it can obtain results using MIRA that match or surpass MERT in terms of both translation quality and computational cost.
Edinburgh Research Explorer (University of Edinburgh)cdec: A Decoder‚ Alignment‚ and Learning framework for finite−state and context−free translation models
230 Citations2010Chris Dyer, Adam Lopez +7 more
Bootstrap estimates for confidence intervals in ASR performance evaluation
223 Citations2004M. Bisani, Hermann Ney
A bootstrap method for significance analysis is presented which is, at the same time, intuitive, precise and and easy to use and immediately interpretable in terms of word error rate.
On Some Pitfalls in Automatic Evaluation and Significance Testing for MT
195 Citations2005Stefan Riezler, John T. Maxwell
In an experimental comparison of two statistical significance tests, it is shown that p-values are estimated more conservatively by approximate randomization than by bootstrap tests, thus increasing the likelihood of type-I error for the latter.
Interpreting BLEU/NIST Scores: How Much Improvement Do We Need to Have a Better System?
165 Citations2004Ying Zhang, Stephan Vogel +1 more
A novel method of calculating the confidence intervals for BLEU/NIST scores using bootstrapping is reported, which can determine whether two MT systems are significantly different from each other.
Using a maximum entropy model to build segmentation lattices for MT
92 Citations2009Chris Dyer
This work describes a maximum entropy model of compound word splitting that relies on a few general features that can be used to generate segmentation lattices for most languages with productive compounding.
Computational LinguisticsWide-Coverage Deep Statistical Parsing Using Automatic Dependency Structure Annotation
84 Citations2008Aoife Cahill, Michael Burke +4 more
This article compares various PCFG and history-based parsers to find a baseline parsing system that fits best into the automatic dependency structure annotation technique, and shows that machine-learning-based shallow grammars augmented with sophisticated automatic dependency annotation technology outperform hand-crafted, deep, wide-coverage constraint Grammars.
Efficient Minimum Error Rate Training and Minimum Bayes-Risk decoding for translation hypergraphs and lattices
81 Citations2009Shankar Kumar, Wolfgang Macherey +2 more
This work extends lattice-based MERT and MBR algorithms to work with hypergraphs that encode a vast number of translations produced by MT systems based on Synchronous Context Free Grammars, and shows how MERT can be employed to optimize parameters for MBR decoding.
The Prague Bulletin of Mathematical LinguisticsImproved Minimum Error Rate Training in Moses
66 Citations2009Nicola Bertoldi, Barry Haddow +1 more
An open-source software for minimum error rate training (MERT) for statistical machine translation (SMT) with the aim of replacing the existing implementation with a cleaner, more flexible design, in order to facilitate further research in weight optimisation.
Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level
61 Citations2010Michael Denkowski, Alon Lavie
It is shown that Meteor-next improves correlation with HTER over baseline metrics, including earlier versions of Meteor, and approaches the correlation level of a state-of-the-art metric, TER-plus (TERp).
The statistical significance of the MUC-5 results
58 Citations1993Nancy Chinchor
The MUC-4 scores of recall, precision, and the F-measures are used to measure the performance of the participating systems and the method of hypothesis testing used is a computationally-intensive method known as approximate randomization.
The Best Lexical Metric for Phrase-Based Statistical MT System Optimization
51 Citations2010Daniel Cer, Christopher D. Manning +1 more
It is shown that people tend to prefer BLEU and NIST trained models to those trained on edit distance based metrics like TER or WER, and that using BLEu or NIST produces models that are more robust to evaluation by other metrics and perform well in human judgments.
Stabilizing minimum error rate training
46 Citations2009George Foster, Roland Kühn
This paper quantifies the stability of Och's procedure by supplying different random seeds to a core component of the procedure (Powell's algorithm) and shows that for systems with many features, there is extensive variation in outcomes, both on the development data and on the test data.
Regularization and search for minimum error rate training
45 Citations2008Daniel Cer, Daniel Jurafsky +1 more
This work contrasts three search strategies for MERT: Powell's method, the variant of coordinate descent found in the Moses MERT utility, and a novel stochastic method.
Random restarts in minimum error rate training for statistical machine translation
40 Citations2008Robert C. Moore, Chris Quirk
It is found that all of the random restart methods outperform MERT withoutrandom restarts, and some refinements of random restarts that are superior to the most common approach with regard to resulting model quality and training time are developed.
Machine TranslationSignificance tests of automatic machine translation evaluation metrics
18 Citations2010Ying Zhang, Stephan Vogel
A significance test-driven comparison of n-gram-based automatic MT evaluation metrics is presented and the characteristics of differentMT evaluation metrics and how to construct reliable and efficient evaluation suites are studied.
Machine TranslationMonte Carlo techniques for phrase-based translation
13 Citations2010Abhishek Arun, Barry Haddow +4 more
A novel Gibbs sampler is defined for sampling translations given a source sentence and it is shown that it effectively explores this posterior distribution of sampling defined by a translation model.