A Systematic Exploration of Diversity in Machine Translation
Published 1 January 2013Open access
Kevin Gimpel, Dhruv Batra, Chris Dyer, Gregory Shakhnarovich
Citations84
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is found that diversity can improve performance on these tasks, especially for sentences that are difficult for MT.
Abstract
This paper addresses the problem of producing a diverse set of plausible translations.We present a simple procedure that can be used with any statistical machine translation (MT) system.We explore three ways of using diverse translations: (1) system combination, (2) discriminative reranking with rich features, and (3) a novel post-editing scenario in which multiple translations are presented to users.We find that diversity can improve performance on these tasks, especially for sentences that are difficult for MT.
Keywords
Computer Science
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
Computational LinguisticsA Systematic Comparison of Various Statistical Alignment Models
3,928 Citations2003Franz Josef Och, Hermann Ney
An important result is that refined alignment models with a first-order dependence and a fertility model yield significantly better results than simple heuristic models.
Statistical phrase-based translation
3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more
The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.
Class-based n -gram models of natural language
2,893 Citations1992Peter F. Brown, P.V. deSouza +3 more
This work addresses the problem of predicting a word from previous words in a sample of text and discusses n-gram models based on classes of words, finding that these models are able to extract classes that have the flavor of either syntactically based groupings or semanticallybased groupings, depending on the nature of the underlying statistics.
Feature-rich part-of-speech tagging with a cyclic dependency network
2,851 Citations2003Kristina Toutanova, Dan Klein +2 more
A new part-of-speech tagger is presented that demonstrates the following ideas: explicit use of both preceding and following tag contexts via a dependency network representation, broad use of lexical features, and effective use of priors in conditional loglinear models.
Computer Speech & LanguageAn empirical study of smoothing techniques for language modeling
2,077 Citations1999Stanley F. Chen, Joshua Goodman
MPG.PuRe (Max Planck Society)Large Margin Methods for Structured and Interdependent Output Variables
1,952 Citations2005Ioannis Tsochantaridis, Thorsten Joachims +2 more
This paper proposes to appropriately generalize the well-known notion of a separation margin and derive a corresponding maximum-margin formulation and presents a cutting plane algorithm that solves the optimization problem in polynomial time for a large class of problems.
Computational LinguisticsHierarchical Phrase-Based Translation
1,164 Citations2007David Chiang
A statistical machine translation model that uses hierarchical phrasesphrases that contain subphrasing that is formally a synchronous context-free grammar but is learned from a parallel text without any syntactic annotations is presented.
KenLM: Faster and Smaller Language Model Queries
1,142 Citations2011Kenneth Heafield
KenLM is a library that implements two data structures for efficient language model queries, reducing both time and memory costs and is integrated into the Moses, cdec, and Joshua translation systems.
Discriminative training and maximum entropy models for statistical machine translation
1,044 Citations2001Franz Josef Och, Hermann Ney
A framework for statistical machine translation of natural languages based on direct maximum entropy models, which contains the widely used source-channel approach as a special case and shows that a baseline statistical machinetranslation system is significantly improved using this approach.
Machine LearningCutting-plane training of structural SVMs
980 Citations2009Thorsten Joachims, Thomas Finley +1 more
This paper explores how cutting-plane methods can provide fast training not only for classification SVMs, but also for structural SVMs and presents an extensive empirical evaluation of the method applied to binary classification, multi-class classification, HMM sequence tagging, and CFG parsing.
Coarse-to-fine <i>n</i>-best parsing and MaxEnt discriminative reranking
888 Citations2005Eugene Charniak, Mark Johnson
This paper describes a simple yet novel method for constructing sets of 50- best parses based on a coarse-to-fine generative parser that generates 50-best lists that are of substantially higher quality than previously obtainable.
IEEE Transactions on Information TheoryThe zero-frequency problem: estimating the probabilities of novel events in adaptive text compression
726 Citations1991Ian H. Witten, Tim Bell
The authors propose the application of a Poisson process model of novelty, which ability to predict novel tokens is evaluated, and it consistently outperforms existing methods and offers a small improvement in the coding efficiency of text compression over the best method previously known.
Discriminative Reranking for Natural Language Parsing
443 Citations2000Michael J. Collins
Computational LinguisticsDiscriminative Reranking for Natural Language Parsing
407 Citations2005Michael J. Collins, Terry Koo
The boosting approach to ranking problems described in Freund et al. (1998) is applied to parsing the Wall Street Journal treebank, and it is argued that the method is an appealing alternative-in terms of both simplicity and efficiency-to work on feature selection methods within log-linear (maximum-entropy) models.
Lattice-based minimum error rate training for statistical machine translation
352 Citations2008Wolfgang Macherey, Franz Josef Och +2 more
A novel algorithm is presented that allows for efficiently constructing and representing the exact error surface of all translations that are encoded in a phrase lattice and is used to train the feature function weights of a phrase-based statistical machine translation system.
Findings of the 2009 workshop on statistical machine translation
351 Citations2009Chris Callison-Burch, Philipp Koehn +2 more
A large-scale manual evaluation of 103 machine translation systems submitted by 34 teams was conducted, which used the ranking of these systems to measure how strongly automatic metrics correlate with human judgments of translation quality for 12 evaluation metrics.
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Minimum Bayes-risk decoding for statistical machine translation
330 Citations2004Shankar Kumar, Bill Byrne
The results show that MBR decoding can be used to tune statistical MT performance for specific loss functions, and a hierarchy of loss functions that incorporate different levels of linguistic information from word strings, word-to-word alignments from an MT system, and syntactic structure from parse-trees of source and target language sentences.
DSpace@MIT (Massachusetts Institute of Technology)Semi-Supervised Learning for Natural Language
304 Citations2005Percy Liang
This thesis focuses on two segmentation tasks, named-entity recognition and Chinese word segmentation, and shows that features derived from unlabeled data substantially improves performance, both in terms of reducing the amount of labeled data needed to achieve a certain performance level and in termsof reducing the error using a fixed amount of labeling data.
Optimizing Chinese word segmentation for machine translation performance
302 Citations2008Pi-Chuan Chang, Michel Galley +1 more
It is demonstrated that optimizing segmentation for an existing segmentation standard does not always yield better MT performance, and it is found that other factors such as segmentation consistency and granularity of Chinese "words" can be more important for machine translation.
Forest Reranking: Discriminative Parsing with Non-Local Features
260 Citations2008Liang Huang
This work presents an approximate algorithm inspired by forest rescoring that makes discriminative training practical over the whole Treebank, and outperforms both 50-best and 100-best reranking baselines and is better than any previously reported systems trained on the Treebank.
North American Chapter of the Association for Computational LinguisticsA Smorgasbord of Features for Statistical Machine Translation
260 Citations2004Franz Josef Och, Daniel Gildea +11 more
A methodology for rapid experimentation in statistical machine translation which is used to add a large number of features to a baseline system exploiting features from a wide range of levels of syntactic representation is described.
ACM Transactions on Mathematical SoftwareObject-oriented software for quadratic programming
255 Citations2003E. Michael Gertz, Stephen J. Wright
Object-oriented software package OOQP for solving convex quadratic programming problems (QP) demonstrates the usefulness of object-oriented design in optimization software development, and establishes standards that can be followed in the design of software packages for other classes of optimization problems.
Tuning as Ranking
246 Citations2011Mark Hopkins, Jonathan May
Pro's scalability and effectiveness is established by comparing it to MERT and MIRA and parity is demonstrated on both phrase-based and syntax-based systems in a variety of language pairs, using large scale data scenarios.
Institutional Research Information System (Università degli Studi di Trento)Findings of the 2013 Workshop on Statistical Machine Translation
243 Citations2013Ondřej Bojar, Christian Buck +8 more
The results of the WMT13 shared tasks, which included a translation task, a task for run-time estimation of machine translation quality, and an unofficial metrics task are presented.
Discriminative Reranking for Machine Translation
207 Citations2004Libin Shen, Anoop Sarkar +1 more
This paper describes the application of discriminative reranking techniques to the problem of machine translation and introduces two novel perceptroninspired reranking algorithms that improve on the quality of machinetranslation over the baseline system based on evaluation using the BLEU metric.
Combining Outputs from Multiple Machine Translation Systems
164 Citations2007Antti-Veikko Rosti, Necip Fazıl Ayan +4 more
The word-level combination provides the most robust gains but the best results on the development test sets (NIST MT05 and the newsgroup portion of GALE 2006 dry-run) were achieved by combining all three methods.
Lecture notes in computer scienceDiverse M-Best Solutions in Markov Random Fields
143 Citations2012Dhruv Batra, Payman Yadollahpour +2 more
This paper proposes an algorithm for the Diverse M-Best problem, which involves finding a diverse set of highly probable solutions under a discrete probabilistic model and shows that for certain families of dissimilarity functions the authors can guarantee that these solutions can be found as easily as the MAP solution.
Exploiting Objective Annotations for Minimising Translation Post-editing Effort
140 Citations2011Lucia Specia
It is shown that estimations resulting from using post-editing time, a simple and objective annotation, can reliably indicate translation post-EDiting effort in a practical, taskbased scenario.
Language model based arabic word segmentation
140 Citations2003Young‐Suk Lee, Kishore Papineni +3 more
This work approximate Arabic's rich morphology by a model that a word consists of a sequence of morphemes in the pattern prefix*-stem-suffix* (* denotes zero or more occurrences of a morpheme).
Lattice Minimum Bayes-Risk decoding for statistical machine translation
127 Citations2008Roy W. Tromble, Shankar Kumar +2 more
Conditions on the loss function that will enable efficient implementation of MBR decoders on lattices are described and an approximation to the BLEU score is introduced that satisfies these conditions.
Solving the problem of cascading errors
115 Citations2006Jenny Rose Finkel, Christopher D. Manning +1 more
A novel architecture is presented, which models these pipelines as Bayesian networks, with each low level task corresponding to a variable in the network, and then it is performed approximate inference to find the best labeling.
Scholarly Commons (University of Pennsylvania)Learning Determinantal Point Processes
108 Citations2012Alex Kulesza, Ben Taskar
A natural feature-based parameterization of conditional DPPs is proposed, and it is shown how it leads to a convex and efficient learning formulation, and is evaluated on data from the DUC 2003/04 multi-document summarization task, achieving state-of-the-art results.
ScholarlyCommons (University of Pennsylvania)Structured Determinantal Point Processes
105 Citations2010Alex Kulesza, Ben Taskar
This model is a marriage of structured probabilistic models, like Markov random fields and context free grammars, with determinantal point processes, which arise in quantum physics as models of particles with repulsive interactions to handle an exponentially-sized set of particles.
Workshop on Statistical Machine TranslationComparing human perceptions of post-editing effort with post-editing operations
98 Citations2012Maarit Koponen
An experiment is described aimed at studying the relationship between technical post-editing effort and cognitive post-EDiting effort by comparing cases where the edit distance and a manual score reflecting perceived effort differ.
Using a maximum entropy model to build segmentation lattices for MT
92 Citations2009Chris Dyer
This work describes a maximum entropy model of compound word splitting that relies on a few general features that can be used to generate segmentation lattices for most languages with productive compounding.
Efficient Minimum Error Rate Training and Minimum Bayes-Risk decoding for translation hypergraphs and lattices
81 Citations2009Shankar Kumar, Wolfgang Macherey +2 more
This work extends lattice-based MERT and MBR algorithms to work with hypergraphs that encode a vast number of translations produced by MT systems based on Synchronous Context Free Grammars, and shows how MERT can be employed to optimize parameters for MBR decoding.
Goodness: A Method for Measuring Machine Translation Confidence
80 Citations2011Nguyễn Bách, Fei Huang +1 more
A novel framework to predict word-level and sentence-level MT errors with a large number of novel features and a visualization prototype of MT errors at the word and sentence levels with the objective to improve post-editor productivity.
Discovering Diverse and Salient Threads in Document Collections
65 Citations2012Jennifer Gillenwater, Alex Kulesza +1 more
This work proposes a novel probabilistic technique for modeling and extracting salient structure from large document collections, particularly interested in revealing and exploiting relationships between documents.
Discriminative Re-ranking of Diverse Segmentations
59 Citations2013Payman Yadollahpour, Dhruv Batra +1 more
This paper introduces a hybrid, two-stage approach to semantic image segmentation, where in the first stage a probabilistic model generates a set of diverse plausible segmentations, and a discriminatively trained re-ranking model selects the best segmentation from this set.
Combination of Machine Translation Systems via Hypothesis Selection from Combined n-best lists
59 Citations2008Almut Silja Hildebrand, Stephan Vogel
This paper presents a method of hypothesis selection which is relatively simple compared to system combination methods which construct a synthesis of the input hypotheses.
Large-Scale Syntactic Language Modeling with Treelets
57 Citations2012Adam Pauls, Dan Klein
A simple generative, syntactic language model that conditions on overlapping windows of tree context (or treelets) in the same way that n-gram language models condition on overlap windows of linear context is proposed, and finds that it performs as well or better than n- gram models and other generative baselines.
An SVM based voting algorithm with application to parse reranking
56 Citations2003Libin Shen, Aravind K. Joshi
A novel Support Vector Machines (SVMs) based voting algorithm for reranking, which provides a way to solve the sequential models indirectly and achieved labeled recall and precision of 89.4%/89.8% on WSJ section 23 of Penn Treebank.
Variational decoding for statistical machine translation
53 Citations2009Zhifei Li, Jason Eisner +1 more
A variational approximation, which considers all the derivations but still allows tractable decoding, is developed, which is similar to minimum-risk decoding for BLEU and improves the state of the art.
An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems
52 Citations2007Wolfgang Macherey, Franz Josef Och
Experimental results are presented for composite translations computed from large numbers of different research systems as well as a set of translation systems derived from one of the bestranked machine translation engines in the 2006 NIST machine translation evaluation.
A Unified Framework for Phrase-Based, Hierarchical, and Syntax-Based Statistical Machine Translation
51 Citations2009Hieu Hoang, Philipp Koehn +1 more
The Moses toolkit is extended to implement hierarchical and syntactic models, making it the first open source toolkit with end-to-end support for all three of these popular models in a single package.
Predicting Machine Translation Adequacy
49 Citations2011Lucia Specia, Najeh Hajlaoui +2 more
The Prague Bulletin of Mathematical LinguisticsCombining Machine Translation Output with Open Source: The Carnegie Mellon Multi-Engine Machine Translation Scheme
48 Citations2010Kenneth Heafield, Alon Lavie
This paper releases the source code behind the combined Arabic-English output with Open Source software, consisting of a recombining beam search decoder, the combination search space and features, and several accessories.
Fast consensus decoding over translation forests
44 Citations2009John DeNero, David Chiang +1 more
This paper introduces and analyze an alternative to MBR that is equally effective at improving performance, yet is asymptotically faster --- running 80 times faster than MBR in experiments with 1000-best lists.
Artificial IntelligenceBagging and Boosting statistical machine translation systems
44 Citations2012Tong Xiao, Jingbo Zhu +1 more
This article addresses the issue of generating diversified translation systems from a single Statistical Machine Translation (SMT) engine for system combination with significant improvements in translation accuracy over the baselines and is able to improve the existing system combination systems.
Correlation between Automatic Evaluation Metric Scores, Post-Editing Speed, and Some Other Factors
35 Citations2009Midori Tatsumi
Wider Pipelines: N-Best Alignments and Parses in MT Training
30 Citations2008Ashish Venugopal, Andreas Zollmann +2 more
This work integrates N-best alignments and parses by using a probability distribution over these alternatives to generate posterior fractional counts for use in downstream estimation, and shows significant improvements in translation quality over a single-best trained baseline.
Minimum Error Rate Training by Sampling the Translation Lattice
22 Citations2010Samidh Chatterjee, Nicola Cancedda
This work empirically measure a systematic improvement in the BLEU scores compared to training using N-best lists, without suffering the increase in computational complexity associated with operating with the whole lattice.
A formal model of ambiguity and its applications in machine translation
20 Citations2010Philip Resnik, Christopher J. Dyer
A model of language processing is advocated in which multiple inputs and multiple analyses of inputs are considered concurrently and a single analysis is only a last resort, and it is shown that translating a set of transcription hypotheses yields better translations compared to a baseline, leading to improved translation quality.
Trait-Based Hypothesis Selection For Machine Translation
17 Citations2012Jacob Devlin, Spyros Matsoukas
A new method for generating diverse hypotheses from a single MT system using traits, which are simple properties of the MT output such as "average output length" and "average rule length."
Positive Diversity Tuning for Machine Translation System Combination
9 Citations2013Daniel Cer, Christopher D. Manning +1 more
Positive Diversity Tuning is presented, a new method for tuning machine translation models specifically for improved performance during system combination that finds that individual systems tuned on the same data to Positive Diversity can be even more diverse than systems built using different data sets, while still obtaining good BLEU scores.
FigshareVoting on N-grams for Machine Translation System Combination
7 Citations2018Kenneth Heafield, Alon Lavie
This work advocates a new feature formulation: for each system and each small n, a feature counts n-gram matches between the system and candidate and shows post-evaluation improvement of 6.67 BLEU over the best system on NIST MT09 Arabic-English test data.