Batch Tuning Strategies for Statistical Machine Translation
NPARCPublished 3 June 2012Open access
Colin Cherry, George Foster
Citations338
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is found that a simple and efficient batch version of MIRA performs at least as well as training online, and consistently outperforms other options.
Abstract
There has been a proliferation of recent work on SMT tuning algorithms capable of han-dling larger feature sets than the traditional MERT approach. We analyze a number of these algorithms in terms of their sentence-level loss functions, which motivates several new approaches, including a Structured SVM. We perform empirical comparisons of eight different tuning strategies, including MERT, in a variety of settings. Among other results, we find that a simple and efficient batch ver-sion of MIRA performs at least as well as training online, and consistently outperforms other options. 1
Keywords
Computer Science
Large-Scale Machine Learning with Stochastic Gradient Descent
5,634 Citations2010Léon Bottou
A more precise analysis uncovers qualitatively different tradeoffs for the case of small-scale and large-scale learning problems.
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
Statistical phrase-based translation
3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more
The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.
Online Passive-Aggressive Algorithms
1,449 Citations2006Koby Crammer, Ofer Dekel +3 more
Support vector machine learning for interdependent and structured output spaces
1,260 Citations2004Ioannis Tsochantaridis, Thomas Hofmann +2 more
This paper proposes to generalize multiclass Support Vector Machine learning in a formulation that involves features extracted jointly from inputs and outputs, and demonstrates the versatility and effectiveness of the method on problems ranging from supervised grammar learning and named-entity recognition, to taxonomic text classification and sequence alignment.
Discriminative training and maximum entropy models for statistical machine translation
1,044 Citations2001Franz Josef Och, Hermann Ney
A framework for statistical machine translation of natural languages based on direct maximum entropy models, which contains the widely used source-channel approach as a special case and shows that a baseline statistical machinetranslation system is significantly improved using this approach.
A dual coordinate descent method for large-scale linear SVM
906 Citations2008Cho‐Jui Hsieh, Kai‐Wei Chang +3 more
A novel dual coordinate descent method for linear SVM with L1-and L2-loss functions that reaches an ε-accurate solution in O(log(1/ε)) iterations is presented.
Learning structural SVMs with latent variables
627 Citations2009Chun-Nam Yu, Thorsten Joachims
A large-margin formulation and algorithm for structured output prediction that allows the use of latent variables and the generality and performance of the approach is demonstrated through three applications including motiffinding, noun-phrase coreference resolution, and optimizing precision at k in information retrieval.
FigshareBetter Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
462 Citations2018Jonathan H. Clark, Chris Dyer +2 more
This paper provides a systematic analysis of the effects of optimizer instability---an extraneous variable that is seldom controlled for---on experimental outcomes, and makes recommendations for reporting results more accurately.
Lattice-based minimum error rate training for statistical machine translation
352 Citations2008Wolfgang Macherey, Franz Josef Och +2 more
A novel algorithm is presented that allows for efficiently constructing and representing the exact error surface of all translations that are encoded in a phrase lattice and is used to train the feature function weights of a phrase-based statistical machine translation system.
An end-to-end discriminative approach to machine translation
265 Citations2006Percy Liang, Alexandre Bouchard‐Côté +2 more
A perceptron-style discriminative approach to machine translation in which large feature sets can be exploited and a novel way to introduce learning into the initial phrase extraction process, which has previously been entirely heuristic.
Distributed Training Strategies for the Structured Perceptron
265 Citations2010Ryan McDonald, Keith Hall +1 more
This paper investigates distributed training strategies for the structured perceptron as a means to reduce training times when computing clusters are available and looks at two strategies and provides convergence bounds for a particular mode of distributed structured perceptrons training based on iterative parameter mixing (or averaging).
Tuning as Ranking
246 Citations2011Mark Hopkins, Jonathan May
Pro's scalability and effectiveness is established by comparing it to MERT and MIRA and parity is demonstrated on both phrase-based and syntax-based systems in a variety of language pairs, using large scale data scenarios.
Online large-margin training of syntactic and structural translation features
242 Citations2008David Chiang, Yuval Marton +1 more
This work explores the use of the MIRA algorithm of Crammer et al. as an alternative to MERT and shows that by parallel processing and exploiting more of the parse forest, it can obtain results using MIRA that match or surpass MERT in terms of both translation quality and computational cost.
11,001 new features for statistical machine translation
238 Citations2009David Chiang, Kevin Knight +1 more
On a large-scale Chinese-English translation task, the Margin Infused Relaxed Algorithm is used to add a large number of new features to two machine translation systems: the Hiero hierarchical phrase-based translation system and the syntax-basedtranslation system.
Discriminative Reranking for Machine Translation
207 Citations2004Libin Shen, Anoop Sarkar +1 more
This paper describes the application of discriminative reranking techniques to the problem of machine translation and introduces two novel perceptroninspired reranking algorithms that improve on the quality of machinetranslation over the baseline system based on evaluation using the BLEU metric.
North American Chapter of the Association for Computational LinguisticsImprovements in Phrase-Based Statistical Machine Translation
186 Citations2004Richard Zens, Hermann Ney
A highly efficient monotone search algorithm with a complexity linear in the input sentence length is described and translated results for three tasks: Verbmobil, Xerox and the Canadian Hansards are presented.
Online Large-Margin Training for Statistical Machine Translation
184 Citations2007Taro Watanabe, Jun Suzuki +2 more
Experiments on Arabic-toEnglish translation indicated that a model trained with sparse binary features outperformed a conventional SMT system with a small number of features.
Minimum risk annealing for training log-linear models
142 Citations2006David A. Smith, Jason Eisner
This work proposes training log-linear combinations of models for dependency parsing and for machine translation, and describes techniques for optimizing nonlinear functions such as precision or the BLEU metric.
First- and second-order expectation semirings with applications to minimum-risk training on translation forests
137 Citations2009Zhifei Li, Jason Eisner
A novel second-order expectation semiring is introduced, which computes second- order statistics and is essential for many interesting training paradigms such as minimum risk, deterministic annealing, active learning, and semi-supervised learning, where gradient descent optimization requires computing the gradient of entropy or risk.
RWTH Publications (RWTH Aachen)Training Phrase Translation Models with Leaving-One-Out
85 Citations2010Joern Wuebker, Arne Mauser +1 more
A novel leaving-one-out approach to prevent over-fitting is described that allows us to train phrase models that show improved translation performance on the WMT08 Europarl German-English task.
FigshareStructured Ramp Loss Minimization for Machine Translation
67 Citations2018Kevin Gimpel, Noah A. Smith
This paper proposes to minimize ramp loss directly and presents a training algorithm that is easy to implement and that performs comparably to others, and is less sensitive to initialization and random seeds than standard approaches.
RWTH Publications (RWTH Aachen)A Systematic Comparison of Training Criteria for Statistical Machine Translation
31 Citations2007Richard Zens, Saša Hasan +1 more
It is shown that, not only from a theoretical point of view but also in terms of translation quality, the minimum Bayes risk decision rule is preferable.
Consensus training for consensus decoding in machine translation
27 Citations2009Adam Pauls, John DeNero +1 more
A novel objective function for discriminatively tuning log-linear machine translation models that specifically optimizes model weights for downstream consensus decoding procedures, and reduces overfitting, which can improve test set BLEU scores when using standard Viterbi decoding.
The Prague Bulletin of Mathematical LinguisticsMargin Infused Relaxed Algorithm for Moses
27 Citations2011Eva Hasler, Barry Haddow +1 more
An open-source implementation of the Margin Infused Relaxed Algorithm for statistical machine translation (SMT) is described and experimental results comparing the performance of MIRA with MERT in terms of translation quality and stability are reported.
Expected BLEU Training for Graphs: BBN System Description for WMT11 System Combination Task
18 Citations2011Antti-Veikko Rosti, Bing Zhang +2 more
System combination weights were tuned using a graph based expected BLEU as the objective function while incrementally expanding the networks to bi-gram and 5-gram contexts and naturally generalizes to hypergraphs and can be used to optimize thousands of weights.
Edinburgh Research Explorer (University of Edinburgh)A Unified Approach to Minimum Risk Training and Decoding
10 Citations2010Abhishek Arun, Barry Haddow +1 more
A unified approach to performing minimum risk training and minimum Bayes risk (MBR) decoding with BLEU in a phrase-based model using a Gibbs sampler that allows us to explore the entire probability distribution and maintain a strict probabilistic formulation across the pipeline.