Fusing Recency into Neural Machine Translation with an Inter-Sentence\n Gate Model
arXiv (Cornell University)Published 12 June 2018Open access
Shaohui Kuang, Deyi Xiong
Citations17
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Neural machine translation (NMT) systems are usually trained on a large\namount of bilingual sentence pairs and translate one sentence at a time,\nignoring inter-sentence information. This may make the translation of a\nsentence ambiguous or even inconsistent with the translations of neighboring\nsentences. In order to handle this issue, we propose an inter-sentence gate\nmodel that uses the same encoder to encode two adjacent sentences and controls\nthe amount of information flowing from the preceding sentence to the\ntranslation of the current sentence with an inter-sentence gate. In this way,\nour proposed model can capture the connection between sentences and fuse\nrecency from neighboring sentences into neural machine translation. On several\nNIST Chinese-English translation tasks, our experiments demonstrate that the\nproposed inter-sentence gate model achieves substantial improvements over the\nbaseline.\n
Keywords
Computer Science
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
arXiv (Cornell University)Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
10,728 Citations2014Jun‐Young Chung, Çaǧlar Gülçehre +2 more
These advanced recurrent units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU), are found to be comparable to LSTM.
Effective Approaches to Attention-based Neural Machine Translation
8,582 Citations2015Thang Luong, Hieu Pham +1 more
A global approach which always attends to all source words and a local one that only looks at a subset of source words at a time are examined, demonstrating the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
6,616 Citations2014Kyunghyun Cho, Bart van Merriënboer +2 more
It is shown that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase.
arXiv (Cornell University)ADADELTA: An Adaptive Learning Rate Method
5,531 Citations2012Matthew D. Zeiler
A novel per-dimension learning rate method for gradient descent called ADADELTA that dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent is presented.
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
2,308 Citations2016William Chan, Navdeep Jaitly +2 more
Listen, Attend and Spell (LAS), a neural speech recognizer that transcribes speech utterances directly to characters without pronunciation models, HMMs or other components of traditional speech recognizers is presented.
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
1,725 Citations2016Iulian Vlad Serban, Alessandro Sordoni +3 more
Improved statistical alignment models
1,015 Citations2000Franz Josef Och, Hermann Ney
It is shown that models with a first-order dependence and a fertility model lead to significantly better results than the simple models IBM-1 or IBM-2, which are not able to go beyond zero-order dependencies.
Edinburgh Research Explorer (University of Edinburgh)Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation
366 Citations2005Philipp Koehn, Amittai Axelrod +4 more
This work adapted the statistical machine translation system that performed successfully in previous DARPA competitions on open domain text translations to work on limited domain speech data in the IWSLT 2005 speech translation task.
Proceedings of the AAAI Conference on Artificial IntelligenceA Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
266 Citations2017Iulian Vlad Serban, Alessandro Sordoni +5 more
A neural network-based generative architecture, with stochastic latent variables that span a variable number of time steps, that improves upon recently proposed models and that the latent variables facilitate both the generation of meaningful, long and diverse responses and maintaining dialogue state is proposed.
Automatic evaluation of text coherence: models and representations
195 Citations2005Mirella Lapata, Regina Barzilay
A fully-automatic, linguistically rich model of local coherence that correlates with human judgments is introduced and demonstrates that certain models capture complementary aspects of coherence and thus can be combined to improve performance.
Hierarchical Recurrent Neural Network for Document Modeling
165 Citations2015Rui Lin, Shujie Liu +4 more
A novel hierarchical recurrent neural network language model (HRNNLM) for document modeling that integrates it as the sentence history information into the word level RNN to predict the word sequence with cross-sentence contextual information.
Transactions of the Association for Computational LinguisticsContext Gates for Neural Machine Translation
108 Citations2017Zhaopeng Tu, Yang Liu +3 more
Context gates are proposed which dynamically control the ratios at which source and target contexts contribute to the generation of target words and can enhance both the adequacy and fluency of NMT with more careful control of the information flow from contexts.
Proceedings of the AAAI Conference on Artificial IntelligenceNeural Machine Translation Advised by Statistical Machine Translation
100 Citations2017Xing Wang, Zhengdong Lu +4 more
Experimental results on Chinese-English translation show that the proposed approach achieves significant and consistent improvements over state-of-the-art NMT and SMT systems on multiple NIST test sets.
Cache-based Document-level Statistical Machine Translation
97 Citations2011Zhengxian Gong, Min Zhang +1 more
This paper presents three kinds of caches to store relevant document-level information: a dynamic cache, which stores bilingual phrase pairs from the best translation hypotheses of previous sentences in the test document; a static cache,which stores relevantilingual phrase pairs extracted from similar bilingual document pairs in the training parallel corpus; and a topic cache,Which stores the target-side topic words related with the test documents in the source-side.
Edinburgh Research ExplorerDocument-Wide Decoding for Phrase-Based Statistical Machine Translation
74 Citations2012Christian Hardmeier, Joakim Nivre +1 more
This work proposes a stochastic local search decoding method for phrase-based SMT, which permits free document-wide dependencies in the models and explores the stability and the search parameters of this method and demonstrates that it can be successfully used to optimise a document-level semantic language model.
Context Adaptation in Statistical Machine Translation Using Models with Exponentially Decaying Cache
63 Citations2010Jörg Tiedemann
An exponential decay factor is applied and cache-based adaptive language and translation models are integrated in a standard phrase-based SMT decoder to obtain a 2.6% relative improvement on average in BLEU scores using the dynamic adaptation procedure.
Memory-enhanced Decoder for Neural Machine Translation
61 Citations2016Mingxuan Wang, Zhengdong Lu +2 more
The memory in this memory-enhanced RNN decoder is a matrix with pre-determined size designed to better capture the information important for the decoding process at each time step, yielding the best performance achieved with the same training set.
Larger-Context Language Modelling with Recurrent Neural Network
57 Citations2016Tian Wang, Kyunghyun Cho
It is found that content words, including nouns, adjec- tives and verbs, benefit most from an increasing number of context sentences, and this analysis suggests that larger-context language model improves the unconditional language model by capturing the theme of a document better and more easily.
IEEE/ACM Transactions on Audio Speech and Language ProcessingTopic-Based Coherence Modeling for Statistical Machine Translation
40 Citations2015Deyi Xiong, Min Zhang +1 more
This paper proposes topic-based coherence models to produce coherence for document translation, in terms of the continuity of sentence topics in a text, and integrates them into a state-of-the-art phrase-based machine translation system.
Proceedings of the AAAI Conference on Artificial IntelligenceA Topic-Based Coherence Model for Statistical Machine Translation
34 Citations2013Deyi Xiong, Min Zhang
This paper proposes a topic-based coherence model to produce coherence for document translation, in terms of the continuity of sentence topics in a text, and adopts a maximum entropy classifier to predict the target coherence chain that defines a linear topic structure for the target document.
Defense Technical Information Center (DTIC)STUDIES IN INTER-SENTENCE CONNECTION
2 Citations1965Kenneth E. Harper
A systematic analysis of portions of the Russian physics text processed at RAND to determine the extent of repetition in adjacent sentences, and the relevance of inter-sentence recurrence and nonrecurrence to automatic syntactic analysis and abstracting is suggested.