Tuning as Ranking
Published 27 July 2011
Mark Hopkins, Jonathan May
Citations246
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Pro's scalability and effectiveness is established by comparing it to MERT and MIRA and parity is demonstrated on both phrase-based and syntax-based systems in a variety of language pairs, using large scale data scenarios.
Abstract
We offer a simple, effective, and scalable method for statistical machine translation pa-rameter tuning based on the pairwise approach to ranking (Herbrich et al., 1999). Unlike the popular MERT algorithm (Och, 2003), our pairwise ranking optimization (PRO) method is not limited to a handful of parameters and can easily handle systems with thousands of features. Moreover, unlike recent approaches built upon the MIRA algorithm of Crammer and Singer (2003) (Watanabe et al., 2007; Chi-ang et al., 2008b), PRO is easy to imple-ment. It uses off-the-shelf linear binary classi-fier software and can be built on top of an ex-isting MERT framework in a matter of hours. We establish PRO’s scalability and effective-ness by comparing it to MERT and MIRA and demonstrate parity on both phrase-based and syntax-based systems in a variety of language pairs, using large scale data scenarios. 1
Keywords
Computer Science
ACM SIGKDD Explorations NewsletterThe WEKA data mining software
17,849 Citations2009Mark Hall, Eibe Frank +4 more
This paper provides an introduction to the WEKA workbench, reviews the history of the project, and, in light of the recent 3.6 stable release, briefly discusses what has been added since the last stable version (Weka 3.4) released in 2003.
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
Learning to rank using gradient descent
2,759 Citations2005Chris Burges, Tal Shaked +5 more
RankNet is introduced, an implementation of these ideas using a neural network to model the underlying ranking function, and test results on toy data and on data from a commercial internet search engine are presented.
A Study of Translation Edit Rate with Targeted Human Annotation
2,402 Citations2006Matthew Snover, Bonnie J. Dorr +3 more
A new, intuitive measure for evaluating machine translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments is defined.
Learning to rank
1,917 Citations2007Zhe Cao, Tao Qin +3 more
It is proposed that learning to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning, and introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning.
Discriminative training and maximum entropy models for statistical machine translation
1,044 Citations2001Franz Josef Och, Hermann Ney
A framework for statistical machine translation of natural languages based on direct maximum entropy models, which contains the widely used source-channel approach as a special case and shows that a baseline statistical machinetranslation system is significantly improved using this approach.
Improved statistical alignment models
1,015 Citations2000Franz Josef Och, Hermann Ney
It is shown that models with a first-order dependence and a fertility model lead to significantly better results than the simple models IBM-1 or IBM-2, which are not able to go beyond zero-order dependencies.
Computational LinguisticsThe Alignment Template Approach to Statistical Machine Translation
933 Citations2004Franz Josef Och, Hermann Ney
A phrase-based statistical machine translation approach the alignment template approach is described, which allows for general many-to-many relations between words and is easier to extend than classical statistical machinetranslation systems.
What's in a Translation Rule
465 Citations2004Michel Galley, Mark Hopkins +2 more
The theory is used to introduce a linear algorithm that can be used to derive from word-aligned, parallel corpora the minimal set of syntactically motivated transformation rules that explain human translation data.
Support vector learning for ordinal regression
458 Citations1999Ralf Herbrich
Experimental results indicate that the presented algorithm outperforms more naive approaches to ordinal regression such as support vector classification and support vector regression in the case of more than two ranks.
Lattice-based minimum error rate training for statistical machine translation
352 Citations2008Wolfgang Macherey, Franz Josef Och +2 more
A novel algorithm is presented that allows for efficiently constructing and representing the exact error surface of all translations that are encoded in a phrase lattice and is used to train the feature function weights of a phrase-based statistical machine translation system.
Machine TranslationThe Meteor metric for automatic evaluation of machine translation
347 Citations2009Alon Lavie, Michael Denkowski
The Meteor Automatic Metric for Machine Translation evaluation, originally developed and released in 2004, was designed with the explicit goal of producing sentence-level scores which correlate well with human judgments of translation quality.
An end-to-end discriminative approach to machine translation
265 Citations2006Percy Liang, Alexandre Bouchard‐Côté +2 more
A perceptron-style discriminative approach to machine translation in which large feature sets can be exploited and a novel way to introduce learning into the initial phrase extraction process, which has previously been entirely heuristic.
Maximum entropy based phrase reordering model for statistical machine translation
254 Citations2006Deyi Xiong, Qun Liu +1 more
A novel reordering model for phrase-based statistical machine translation (SMT) that uses a maximum entropy (MaxEnt) model to predicate reorderings of neighbor blocks (phrase pairs) that obtains significant improvements in BLEU score on the NIST MT-05 and IWSLT-04 tasks.
Online large-margin training of syntactic and structural translation features
242 Citations2008David Chiang, Yuval Marton +1 more
This work explores the use of the MIRA algorithm of Crammer et al. as an alternative to MERT and shows that by parallel processing and exploiting more of the parse forest, it can obtain results using MIRA that match or surpass MERT in terms of both translation quality and computational cost.
11,001 new features for statistical machine translation
238 Citations2009David Chiang, Kevin Knight +1 more
On a large-scale Chinese-English translation task, the Margin Infused Relaxed Algorithm is used to add a large number of new features to two machine translation systems: the Hiero hierarchical phrase-based translation system and the syntax-basedtranslation system.
Precision and recall of machine translation
227 Citations2003I. Dan Melamed, Ryan Green +1 more
Machine translation can be evaluated using precision, recall, and the F-measure, which have significantly higher correlation with human judgments than recently proposed alternatives.
Discriminative Reranking for Machine Translation
207 Citations2004Libin Shen, Anoop Sarkar +1 more
This paper describes the application of discriminative reranking techniques to the problem of machine translation and introduces two novel perceptroninspired reranking algorithms that improve on the quality of machinetranslation over the baseline system based on evaluation using the BLEU metric.
Online Large-Margin Training for Statistical Machine Translation
184 Citations2007Taro Watanabe, Jun Suzuki +2 more
Experiments on Arabic-toEnglish translation indicated that a model trained with sparse binary features outperformed a conventional SMT system with a small number of features.
Optimization, maxent models, and conditional estimation without magic
150 Citations2003Christopher D. Manning, Dan Klein
This tutorial aims to cover the basic ideas and algorithms behind techniques such as maximum entropy modeling, conditional estimation of generative probabilistic models, and issues regarding the use of models more complex than simple Naive Bayes and Hidden Markov Models, using graphical illustrations rather than detailed derivations whenever possible.
Ranking Measures and Loss Functions in Learning to Rank
149 Citations2009Wei Chen, Tie‐Yan Liu +3 more
The relationship between ranking measures and loss functions in learningto-rank methods, such as Ranking SVM, RankBoost, RankNet, and ListMLE are revealed, and it is proved that the essential loss is both an upper bound of the measure-based ranking errors, and a lower Bound of the loss functions of the aforementioned methods.
A maximum entropy word aligner for Arabic-English machine translation
115 Citations2005Abraham Ittycheriah, Salim Roukos
It is demonstrated that it is feasible to create training material for problems in machine translation and that a mixture of supervised and unsupervised methods yields superior performance.
Learning to Translate with Source and Target Syntax
109 Citations2010David Chiang
This work discusses the reasons why this has been a challenge, reviews existing attempts to meet this challenge, and shows how some old and new ideas can be combined into a simple approach that uses both source and target syntax for significant improvements in translation accuracy.
Efficient Minimum Error Rate Training and Minimum Bayes-Risk decoding for translation hypergraphs and lattices
81 Citations2009Shankar Kumar, Wolfgang Macherey +2 more
This work extends lattice-based MERT and MBR algorithms to work with hypergraphs that encode a vast number of translations produced by MT systems based on Synchronous Context Free Grammars, and shows how MERT can be employed to optimize parameters for MBR decoding.
A discriminative model for tree-to-tree translation
77 Citations2006Brooke Cowan, Ivona Kučerová +1 more
A statistical, tree-to-tree model for producing translations with use of a discriminative, feature-based model for prediction of target-language syntactic structures---which the authors call aligned extended projections, or AEPs.
A localized prediction model for statistical machine translation
65 Citations2005Christoph Tillmann, Tong Zhang
A novel training method for a localized phrase-based prediction model for statistical machine translation (SMT) that predicts blocks with orientation to handle local phrase re-ordering and uses a maximum likelihood criterion to train a log-linear block bigram model.
Decomposability of translation metrics for improved evaluation and efficient algorithms
49 Citations2008David Chiang, Steve DeNeefe +2 more
This work proposes a very conservative modification toBleu and a cross between Bleu and word error rate that address issues while improving correlation with human judgments.
Minimum Error Rate Training by Sampling the Translation Lattice
22 Citations2010Samidh Chatterjee, Nicola Cancedda
This work empirically measure a systematic improvement in the BLEU scores compared to training using N-best lists, without suffering the increase in computational complexity associated with operating with the whole lattice.
NTT statistical machine translation for IWSLT 2006.
16 Citations2006Taro Watanabe, Jun Suzuki +2 more
Experiments showed that the hierarchical phrase-based model outperformed a conventional phrase- based model and the reranking algorithm further boosted the performance.
Conference of the Association for Machine Translation in the AmericasMachine Translation Using Overlapping Alignments and SampleRank
4 Citations2009Benjamin Roth, Andrew McCallum +2 more
A conditional-random-field approach to discriminatively-trained phrase-based machine translation in which training and decoding are both cast in a sampling framework and are implemented uniformly in a new probabilistic programming language for factor graphs.