Bilingual Word Embeddings for Phrase-Based Machine Translation
Published 1 January 2013Open access
Will Y. Zou, Richard Socher, Daniel Cer, Christopher D. Manning
Citations542
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A method to learn bilingual embeddings from a large unlabeled corpus, while utilizing MT word alignments to constrain translational equivalence is proposed, which significantly out-perform baselines in word semantic similarity.
Abstract
We introduce bilingual word embeddings: se-mantic embeddings associated across two lan-guages in the context of neural language mod-els. We propose a method to learn bilingual embeddings from a large unlabeled corpus, while utilizing MT word alignments to con-strain translational equivalence. The new em-beddings significantly out-perform baselines in word semantic similarity. A single semantic similarity feature induced with bilingual em-beddings adds near half a BLEU point to the results of NIST08 Chinese-English machine translation task. 1
Keywords
Computer Science
Journal of Machine Learning ResearchVisualizing Data using t-SNE
35,654 Citations2008Laurens van der Maaten, Geoffrey E. Hinton
A new technique called t-SNE that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map, a variation of Stochastic Neighbor Embedding that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
11,710 Citations2013Tomáš Mikolov, Kai Chen +2 more
ACM Computing SurveysMachine learning in automated text categorization
7,899 Citations2002Fabrizio Sebastiani
This survey discusses the main approaches to text categorization that fall within the machine learning paradigm and discusses in detail issues pertaining to three different problems, namely, document representation, classifier construction, and classifier evaluation.
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
Curriculum learning
4,978 Citations2009Yoshua Bengio, Jérôme Louradour +2 more
It is hypothesized that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).
Learning Word Vectors for Sentiment Analysis
3,299 Citations2011Andrew L. Maas, Raymond E. Daly +4 more
This work presents a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term--document information as well as rich sentiment content, and finds it out-performs several previously introduced methods for sentiment classification.
Statistical phrase-based translation
3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more
The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
Word Representations: A Simple and General Method for Semi-Supervised Learning
1,944 Citations2010Joseph Turian, Lev-Arie Ratinov +1 more
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines.
Deep Neural Networks for Acoustic Modeling in Speech Recognition
1,899 Citations2012Geoffrey E. Hinton, Li Deng +9 more
This paper provides an overview of this progress and repres nts the shared views of four research groups who have had recent successes in using deep neural networks for a coustic modeling in speech recognition.
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
1,198 Citations2011Richard Socher, Jeffrey Pennington +3 more
A novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions that outperform other state-of-the-art approaches on commonly used datasets, without using any pre-defined sentiment lexica or polarity shifting rules.
Improving Word Representations via Global Context and Multiple Word Prototypes
1,101 Citations2012Eric Huang, Richard Socher +2 more
A new neural network architecture is presented which learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and accounts for homonymy and polysemy by learning multiple embedDings per word.
Scaling learning algorithms towards AI
927 Citations2007Yoshua Bengio, Yann LeCun
It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.
A Scalable Hierarchical Distributed Language Model
852 Citations2008Andriy Mnih, Geoffrey E. Hinton
A fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data are introduced and it is shown that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models.
International Conference on Artificial Intelligence and StatisticsHierarchical Probabilistic Neural Network Language Model.
840 Citations2005Fréderic Morin, Yoshua Bengio
A point-of-care POC analyzer that can rapidly measure both medications used for treatment and illicit drugs in patient saliva and shows evidence of accuracy in measuring treatment medications and relapse to drug use is shown.
Conference on Computational Natural Language LearningBetter Word Representations with Recursive Neural Networks for Morphology
820 Citations2013Thang Luong, Richard Socher +1 more
This paper combines recursive neural networks, where each morpheme is a basic unit, with neural language models to consider contextual information in learning morphologicallyaware word representations and proposes a novel model capable of building representations for morphologically complex words from their morphemes.
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
810 Citations2011Richard Socher, Eric Huang +3 more
This work introduces a method for paraphrase detection based on recursive autoencoders (RAE) and unsupervised RAEs based on a novel unfolding objective and learns feature vectors for phrases in syntactic trees to measure word- and phrase-wise similarity between two sentences.
A syntax-based statistical translation model
736 Citations2001Kenji Yamada, Kevin Knight
This model transforms a source-language parse tree into a target-language string by applying stochastic operations at each node, and produces word alignments that are better than those produced by IBM Model 5.
Generating query substitutions
614 Citations2006Rosie Jones, Benjamin Rey +2 more
A model for selecting between candidates is built, by using a number of features relating the query-candidate pair, and by fitting the model to human judgments of relevance of query suggestions, which improves the quality of the candidates generated.
Alignment by agreement
433 Citations2006Percy Liang, Ben Taskar +1 more
An unsupervised approach to symmetric word alignment in which two simple asymmetric models are trained jointly to maximize a combination of data likelihood and agreement between the models.
Inducing Crosslingual Distributed Representations of Words
373 Citations2012Alexandre Klementiev, Ivan Titov +1 more
This work induces distributed representations for a pair of languages jointly and shows that these representations are informative by using them for crosslingual document classification, where classifiers trained on these representations substantially outperform strong baselines when applied to a new language.
Multi-Prototype Vector-Space Models of Word Meaning
355 Citations2010Joseph Reisinger, Raymond J. Mooney
Experimental comparisons to human judgements of semantic similarity for both isolated words as well as words in sentential contexts demonstrate the superiority of this approach over both prototype and exemplar based vector-space models.
Quantitative evaluation of passage retrieval algorithms for question answering
284 Citations2003Stefanie Tellex, Boris Katz +3 more
This work presents a quantitative evaluation of various passage retrieval algorithms for question answering, implemented in a framework called Pauchok, and presents three important findings: Boolean querying schemes perform well in the question answering task.
Holistic Sentiment Analysis Across Languages: Multilingual Supervised Latent Dirichlet Allocation
133 Citations2010Jordan Boyd‐Graber, Philip Resnik
This paper develops multilingual supervised latent Dirichlet allocation (MlSLDA), a probabilistic generative model that allows insights gleaned from one language's data to inform how the model captures properties of other languages.
HAL (Le Centre pour la Communication Scientifique Directe)Continuous space translation models with neural networks
122 Citations2012Hai-Son Le, Alexandre Allauzen +1 more
Several continuous space translation models are explored, where translation probabilities are estimated using a continuous representation of translation units in lieu of standard discrete representations, jointly computed using a multi-layer neural network with a SOUL architecture.
Names and similarities on the web
83 Citations2006Marius Paşca, Dekang Lin +3 more
In a new approach to large-scale extraction of facts from unstructured text, distributional similarities become an integral part of both the iterative acquisition of high-coverage contextual extraction patterns and the validation and ranking of candidate facts.
The Best Lexical Metric for Phrase-Based Statistical MT System Optimization
51 Citations2010Daniel Cer, Christopher D. Manning +1 more
It is shown that people tend to prefer BLEU and NIST trained models to those trained on edit distance based metrics like TER or WER, and that using BLEu or NIST produces models that are more robust to evaluation by other metrics and perform well in human judgments.
SemEval-2012 Task 4: Evaluating Chinese Word Similarity
48 Citations2012Peng Jin, Yunfang Wu
This task focuses on evaluating word similarity computation in Chinese by organizing twenty undergraduates who are major in Chinese linguistics to annotate the data and ranking the word pairs by the average value of similar scores among the twenty annotators.
Stabilizing minimum error rate training
46 Citations2009George Foster, Roland Kühn
This paper quantifies the stability of Och's procedure by supplying different random seeds to a core component of the procedure (Powell's algorithm) and shows that for systems with many features, there is extensive variation in outcomes, both on the development data and on the test data.
Joint Word Alignment and Bilingual Named Entity Recognition Using Dual Decomposition
40 Citations2013Mengqiu Wang, Wanxiang Che +1 more
A graphical model is presented that performs bilingual NER tagging jointly with word alignment, by combining two monolingual tagging models with two unidirectional alignment models, and a dual decomposition inference algorithm is designed to perform joint decoding over the combined alignment and NER output space.
Fast and Adaptive Online Training of Feature-Rich Translation Models
36 Citations2013Spence Green, Sida Wang +2 more
This work presents a fast and scalable online method for tuning statistical machine translation models with large feature sets, which scales to millions of features and tuning sets with tens of thousands of sentences, while still converging after only a few epochs.
Phrasal: a toolkit for statistical machine translation with facilities for extraction and incorporation of arbitrary model features
32 Citations2010Daniel Cer, Michel Galley +2 more
A new Java-based open source toolkit for phrase-based machine translation to use APIs for integrating new features (/knowledge sources) into the decoding model and for extracting feature statistics from aligned bitexts is presented.
Lirias (KU Leuven)Cross-lingual Induction of Selectional Preferences with Bilingual Vector Spaces
30 Citations2010Yves Peirsman, Sebastian Padó
A cross-lingual method for the induction of selectional preferences for resource-poor languages, where no accurate monolingual models are available, and the predictions correlate well with human ratings, clearly outperformingmonolingual baseline models.
Proceedings of the Fourth Workshop on Statistical Machine Translation
29 Citations2009Chris Callison-Burch, Philipp Koehn +2 more
Bilingual-LSA Based LM Adaptation for Spoken Language Translation
27 Citations2007Wilson Tam, Ian Lane +1 more
A bLSA model is introduced which enables latent topic distributions to be efficiently transferred across languages by enforcing a one-to-one topic correspondence during training and consistently improved machine translation quality on both speech and text based adaptation.
Lexical transfer using a vector-space model
7 Citations2000Eiichiro Sumita
A new mechanism for lexical transfer is proposed, which is simple and suitable for learning from bilingual corpora and exploits a vector-space model developed in information retrieval research.
NIST Open Machine Translation 2008 Evaluation: Stanford University's System Description
3 Citations2008Michel Galley, Pi-Chuan Chang +3 more
This document describes Stanford University’s first entry into a NIST MT evaluation, establishing a competent baseline with a phrase-based system similar to (Och and Ney, 2004; Koehn et al., 2007).