Measuring the Influence of Long Range Dependencies with Neural Network Language Models
North American Chapter of the Association for Computational LinguisticsPublished 8 June 2012
Hai-Son Le, Alexandre Allauzen, François Yvon
Citations26
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is experimentally observed that extending the context size yields clear gains in terms of perplexity and that the n-gram assumption is statistically reasonable as long as n is sufficiently high, and that efforts should be focused on improving the estimation procedures for such large models.
Abstract
In spite of their well known limitations, most notably their use of very local contexts, n-gram language models remain an essential component of many Natural Language Processing applications, such as Automatic Speech Recognition or Statistical Machine Translation. This paper investigates the potential of language models using larger context windows comprising up to the 9 previous words. This study is made possible by the development of several novel Neural Network Language Model architectures, which can easily fare with such large context windows. We experimentally observed that extending the context size yields clear gains in terms of perplexity and that the n-gram assumption is statistically reasonable as long as n is sufficiently high, and that efforts should be focused on improving the estimation procedures for such large models.
Keywords
Computer Science
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Class-based n -gram models of natural language
2,893 Citations1992Peter F. Brown, P.V. deSouza +3 more
This work addresses the problem of predicting a word from previous words in a sample of text and discusses n-gram models based on classes of words, finding that these models are able to extract classes that have the flavor of either syntactically based groupings or semanticallybased groupings, depending on the nature of the underlying statistics.
Minimum error rate training in statistical machine translation
2,766 Citations2003Franz Josef Och
It is shown that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
Applied Physics Letters10.1162/153244303322533223
1,771 Citations2000
This work proposes to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.
Extensions of recurrent neural network language model
1,596 Citations2011Tomáš Mikolov, Stefan Kombrink +3 more
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model.
A Scalable Hierarchical Distributed Language Model
852 Citations2008Andriy Mnih, Geoffrey E. Hinton
A fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data are introduced and it is shown that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models.
Proceedings of the IEEETwo decades of statistical language modeling: where do we go from here?
726 Citations2000Roni Rosenfeld
A Bayesian approach to integration of linguistic theories with data is argued for inStatistical language models estimate the distribution of various natural language phenomena for the purpose of speech recognition and other language technologies.
ArXiv.orgAn Empirical Study of Smoothing Techniques for Language Modeling
633 Citations1996Stanley F. Chen, Joshua Goodman
Large Language Models in Machine Translation
550 Citations2007Thorsten Brants, Ashok C. Popat +3 more
Systems, methods, and computer program products for machine translation are provided for backoff score determination as a function of a backoff factor and a relative frequency of a corresponding backoff n-gram in the corpus.
Strategies for training large scale neural network language models
512 Citations2011Tomáš Mikolov, Anoop Deoras +3 more
This work describes how to effectively train neural network based language models on large data sets and introduces hash-based implementation of a maximum entropy model, that can be trained as a part of the neural network model.
Computer Speech & LanguageContinuous space language models
503 Citations2006Holger Schwenk
Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words and it is shown that this approach can be incorporated into a large vocabulary continuous speech recognizer using a lattice rescoring framework at a very low additional processing time.
IEEE Transactions on Pattern Analysis and Machine IntelligenceA cache-based natural language model for speech recognition
503 Citations1990Roland Kühn, Renato De Mori
A novel kind of language model which reflects short-term patterns of word use by means of a cache component (analogous to cache memory in hardware terminology) is presented and contains a 3g-gram component of the traditional type.
A hierarchical Bayesian language model based on Pitman-Yor processes
498 Citations2006Yee Whye Teh
It is shown that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models.
Computer Speech & LanguageStructured language modeling
275 Citations2000Ciprian Chelba, Frederick Jelinek
An attempt at using the syntactic structure in natural language for improved language models for speech recognition using an original probabilistic parameterization of a shift-reduce parser.
Structured Output Layer neural network language model
131 Citations2011Hai-Son Le, Ilya Oparin +3 more
A new neural network language model (NNLM) based on word clustering to structure the output vocabulary: Structured Output Layer NNLM, able to handle vocabularies of arbitrary size, hence dispensing with the design of short-lists that are commonly used in NNLMs.
IEEE International Conference on Acoustics Speech and Signal ProcessingConnectionist language modeling for large vocabulary continuous speech recognition
116 Citations2002Holger Schwenk, Jean‐Luc Gauvain
The connectionist language model is being evaluated on the DARPA HUB5 conversational telephone speech recognition task and preliminary results show consistent improvements in both perplexity and word error rate.
Using story topics for language model adaptation
104 Citations1997Kristie Seymore, Roni Rosenfeld
A language model adaptation scheme that takes a piece of text, chooses the most similar topic clusters from a set of over 5000 elemental topics, and uses topic specific language models built from the topic clusters to rescore N-best lists, allowing for adaptation to unique, previously unseen combinations of subjects.
Computer Speech & LanguageWhole-sentence exponential language models: a vehicle for linguistic-statistical integration
93 Citations2001Ronald Rosenfeld, Stanley F. Chen +1 more
An exponential language model which models a whole sentence or utterance as a single unit is introduced, and a novel procedure for feature selection is presented, which exploits discrepancies between the existing model and the training corpus.
Discriminative syntactic language modeling for speech recognition
85 Citations2005Michael Collins, Brian Roark +1 more
A reranking model makes use of syntactic features together with a parameter estimation method that is based on the perception algorithm that provides an additional 0.3% reduction in test-set error rate beyond the model of (Roark et al., 2004a; Roark etAl., 2004b).
Adaptive language modeling using the maximum entropy principle
57 Citations1993Raymond Y.K. Lau, Roni Rosenfeld +1 more
A succession of ME models are described, culminating in the current Maximum Likelihood/Maximum Entropy (ML/ME) model, which shows a 27% perplexity reduction as compared to a conventional trigram model.
Elsevier eBooksInternational Speech Communication Association
49 Citations2006Sadaoki Furui
The International Speech Communication Association (ISCA) is a worldwide nonprofit organization that supports research in all aspects of spoken-language communication.
Edinburgh Research Explorer (University of Edinburgh)Large and Diverse Language Models for Statistical Machine Translation
38 Citations2009Holger Schwenk, Philipp Koehn
Methods to combine large language models trained from diverse text sources and applies them to a state-ofart French–English and Arabic–English machine translation system are presented.
Incremental Syntactic Language Models for Phrase-based Translation
28 Citations2011Lane Schwartz, Chris Callison-Burch +2 more
Large vocabulary SOUL neural network language models
24 Citations2011Hai Son Le, Ilya Oparin +4 more
A new training scheme is proposed for SOUL NNLMs that is based on separate training of the outof-shortlist part of the output layer, which enables using more data at each iteration of a neural network without any considerable slow-down in training and brings additional improvements in speech recognition performance.
Morphological and syntactic features for Arabic speech recognition
14 Citations2010Hong-Kwang Jeff Kuo, Lidia Mangu +2 more
Using morphological and syntactic features, including part-of-speech tags, shallow parse tags, and exposed head words and their non-terminal labels both before and after the word to be predicted, can improve the word error rate significantly on various test sets.
Improving LVCSR system combination using neural network language model cross adaptation
11 Citations2011Xiaobing Liu, Mark Gales +1 more
Previous research on multi-level n-gram LM cross adaptation is extended to further include the cross adaptation of neural network LMs in this paper, and significant error rate gains were obtained when combining a range of Chinese LVCSR sub-systems used in the 2010 and 2011 DARPA GALE evaluations.
LIMSI @ WMT11
9 Citations2011Alexandre Allauzen, Hélène Bonneau-Maynard +9 more
This paper describes LIMSI's submissions to the Sixth Workshop on Statistical Machine Translation, and investigates the use of the SOUL language model in Machine Translation and shows significant improvements with a 10-gram SOUL model.