LSTM based Conversation Models
arXiv (Cornell University)Published 31 March 2016Open access
Yi Luan, Yangfeng Ji, Mari Ostendorf
Citations58
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A conversational model that incorporates both context and participant role for two-party conversations that outperforms a traditional LSTM model as measured by language model perplexity and response ranking is presented.
Abstract
In this paper, we present a conversational model that incorporates both context and participant role for two-party conversations. Different architectures are explored for integrating participant role and context information into a Long Short-term Memory (LSTM) language model. The conversational model can function as a language model or a language generation model. Experiments on the Ubuntu Dialog Corpus show that our model can capture multiple turn interaction between participants. The proposed method outperforms a traditional LSTM model as measured by language model perplexity and response ranking. Generated responses show characteristic differences between the two participant roles.
Keywords
Computer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
Machine learning: a probabilistic perspective
9,297 Citations2012Kevin P. Murphy
Software Framework for Topic Modelling with Large Corpora
3,806 Citations2010R Rehr Uv Rek, Petr Sojka
This work describes a Natural Language Processing software framework which is based on the idea of document streaming, i.e. processing corpora document after document, in a memory independent fashion, and implements several popular algorithms for topical inference, including Latent Semantic Analysis and Latent Dirichlet Allocation in a way that makes them completely independent of the training corpus size.
arXiv (Cornell University)A Neural Conversational Model
1,500 Citations2015Oriol Vinyals, Quoc V. Le
A simple approach to conversational modeling which uses the recently proposed sequence to sequence framework, and is able to extract knowledge from both a domain specific dataset, and from a large, noisy, and general domain dataset of movie subtitles.
Neural Responding Machine for Short-Text Conversation
1,046 Citations2015Lifeng Shang, Zhengdong Lu +1 more
Empirical study shows that NRM can generate grammatically correct and content-wise appropriate responses to over 75% of the input text, outperforming state-of-the-arts in the same setting, including retrieval-based and SMT-based models.
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
846 Citations2015Tsung-Hsien Wen, Milica Gašić +4 more
A statistical language generator based on a semantically controlled Long Short-term Memory (LSTM) structure that can learn from unaligned data by jointly optimising sentence planning and surface realisation using a simple cross entropy training criterion, and language variation can be easily achieved by sampling from output candidates.
FigshareImproved Part-of-Speech Tagging for Online Conversational Text with Word Clusters
682 Citations2018Olutobi Owoputi, Brendan O’Connor +4 more
This work systematically evaluates the use of large-scale unsupervised word clustering and new lexical features to improve tagging accuracy on Twitter and achieves state-of-the-art tagging results on both Twitter and IRC POS tagging tasks.
National Research Council Canada (Government of Canada)Data-Driven Response Generation in Social Media
576 Citations2011Alan Ritter, Colin Cherry +1 more
It is found that mapping conversational stimuli onto responses is more difficult than translating between languages, due to the wider range of possible responses, the larger fraction of unaligned words/phrases, and the presence of large phrase pairs whose alignment cannot be further decomposed.
Context dependent recurrent neural network language model
574 Citations2012Tomáš Mikolov, Geoffrey Zweig
This paper improves recurrent neural network language models performance by providing a contextual real-valued input vector in association with each word to convey contextual information about the sentence being modeled by performing Latent Dirichlet Allocation using a block of preceding text.
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
227 Citations2015Ryan Lowe, Nissan Pow +2 more
The Ubuntu Dialogue Corpus is introduced, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words, that provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data.
A Dataset for Research on Short-Text Conversations
191 Citations2013Hao Wang, Zhengdong Lu +2 more
This paper introduces a dataset of short-text conversation based on the real-world instances from Sina Weibo, which provides rich collection of instances for the research on finding natural and relevant short responses to a given short text, and useful for both training and testing of conversation models.
arXiv (Cornell University)A Persona-Based Neural Conversation Model
181 Citations2016Jiwei Li, Michel Galley +4 more
Hierarchical Recurrent Neural Network for Document Modeling
165 Citations2015Rui Lin, Shujie Liu +4 more
A novel hierarchical recurrent neural network language model (HRNNLM) for document modeling that integrates it as the sentence history information into the word level RNN to predict the word sequence with cross-sentence contextual information.
arXiv (Cornell University)deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
93 Citations2015Michel Galley, Chris Brockett +7 more
Columbia Academic Commons (Columbia University)The Rules Behind Roles: Identifying speaker role in radio broadcasts
92 Citations2000Regina Barzilay, Michael J. Collins +2 more
An algorithm is implemented that classies story segments into three Speaker Roles based on several content and duration features and correctly classies about 80% of segments when applied to ASR derived transcriptions of broadcast data.
arXiv (Cornell University)A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
78 Citations2015Alessandro Sordoni, Michel Galley +7 more
arXiv (Cornell University)Document Context Language Models
60 Citations2015Yangfeng Ji, Trevor Cohn +3 more
Unsupervised broadcast conversation speaker role labeling
29 Citations2010Brian Hutchinson, Bin Zhang +1 more
An approach to unsupervised speaker role labeling in talk show data is presented that makes use of two complementary sets of features: structural features that encode the participation patterns of speakers, and lexical features, which capture characteristic phrases.
Multi-speaker language modeling
27 Citations2004Gang Ji, Jeff Bilmes
By augmenting a normal trigram context, this work introduces an approach for conversational language modeling that considers words from other speakers when predicting words from the current one and achieves a 8.9% perplexity reduction on Switchboard and a 12.2% reduction on the ICSI Meeting Recorder data.
Detecting Forum Authority Claims in Online Discussions
17 Citations2011Alex Marin, Bin Zhang +1 more
It is found that certain markup features perform remarkably well alone, but are outperformed by data-driven selection of lexical features augmented with parse context.
IEEE/ACM Transactions on Audio Speech and Language ProcessingA Sparse Plus Low-Rank Exponential Language Model for Limited Resource Scenarios
13 Citations2015Brian Hutchinson, Mari Ostendorf +1 more
A new exponential language model that decomposes the model parameters into one or more low-rank matrices that learn regularities in the training data and oneor more sparse matrices That learn exceptions (e.g., keywords) is described.
Lecture notes in computer scienceModeling Topic and Role Information in Meetings Using the Hierarchical Dirichlet Process
7 Citations2008Songfang Huang, Steve Renals
This paper presents a modeling framework for topic and role on the AMI Meeting Corpus, and illustrates the effectiveness of the approach in the context of adapting a baseline language model in a large-vocabulary automatic speech recognition system for multiparty meetings.
ProQuest LLC eBooksRank and sparsity in language processing
4 Citations2013Mari Ostendorf, Maryam Fazel +1 more
This thesis demonstrates that sparsity acts as a complement to the low rank parameters: a low rank component learns the regularities that exist in language, while a sparse one captures the exceptional sequence phenomena, making the models useful not only for computing probabilities but as tools for the analysis of language.