What makes a good conversation? How controllable attributes affect human judgments
Published 1 January 2019Open access
Abigail See, Stephen Roller, Douwe Kiela, Jason Weston
Citations226
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work examines two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chit-chat dialogue: repetition, specificity, response-relatedness and question-asking, and shows that by controlling combinations of these variables their models obtain clear improvements in human quality judgments.
Abstract
Abigail See, Stephen Roller, Douwe Kiela, Jason Weston. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Keywords
Computer Science
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
2,035 Citations2015Yukun Zhu, Ryan Kiros +5 more
To align movies and books, a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book are proposed.
A Diversity-Promoting Objective Function for Neural Conversation Models
1,986 Citations2016Jiwei Li, Michel Galley +3 more
This work proposes using Maximum Mutual Information (MMI) as the objective function in neural models, and demonstrates that the proposed MMI models produce more diverse, interesting, and appropriate responses, yielding substantive gains in BLEU scores on two conversational datasets and in human evaluations.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
1,146 Citations2018Saizheng Zhang, Emily Dinan +4 more
This work collects data and train models tocondition on their given profile information; and information about the person they are talking to, resulting in improved dialogues, as measured by next utterance prediction.
International Conference on Learning RepresentationsA Simple but Tough-to-Beat Baseline for Sentence Embeddings
1,054 Citations2017Sanjeev Arora, Yingyu Liang +1 more
A Network-based End-to-End Trainable Task-oriented Dialogue System
806 Citations2017Tsung-Hsien Wen, David Vandyke +6 more
This work introduces a neural network-based text-in, text-out end-to-end trainable goal-oriented dialogue system along with a new way of collecting dialogue data based on a novel pipe-lined Wizard-of-Oz framework that can converse with human subjects naturally whilst helping them to accomplish tasks in a restaurant search domain.
Adversarial Learning for Neural Dialogue Generation
765 Citations2017Jiwei Li, Will Monroe +4 more
This work applies adversarial training to open-domain dialogue generation, training a system to produce sequences that are indistinguishable from human-generated dialogue utterances, and investigates models for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls.
arXiv (Cornell University)How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
613 Citations2016Chia‐Wei Liu, Ryan Lowe +4 more
The Second Dialog State Tracking Challenge
580 Citations2014Matthew Henderson, Blaise Thomson +1 more
The results suggest that while large improvements on a competitive baseline are possible, trackers are still prone to degradation in mismatched conditions and ensemble learning demonstrates the most accurate tracking can be achieved by combining multiple trackers.
PARADISE
538 Citations1997Marilyn Walker, Diane Litman +2 more
This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating spoken dialogue agents that decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, and makes it possible to compare agents performing different tasks by normalizing for task complexity.
arXiv (Cornell University)Wizard of Wikipedia: Knowledge-Powered Conversational agents
479 Citations2018Emily Dinan, Stephen Roller +4 more
The Springer series on challenges in machine learningThe Second Conversational Intelligence Challenge (ConvAI2)
362 Citations2019Emily Dinan, Varvara Logacheva +15 more
To improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations)—in terms of repetition, consistency and balance of dialogue acts.
Proceedings of the AAAI Conference on Artificial IntelligenceTopic Aware Neural Response Generation
321 Citations2017Xing Chen, Wei Wu +5 more
A topic aware sequence-to-sequence (TA-Seq2Seq) model that utilizes topics to simulate prior human knowledge that guides them to form informative and interesting responses in conversation, and leverages topic information in generation by a joint attention mechanism and a biased generation probability.
arXiv (Cornell University)Topic Aware Neural Response Generation
317 Citations2016Xing Chen, Wei Wu +5 more
Controllable Abstractive Summarization
273 Citations2018Angela Fan, David Grangier +1 more
A neural summarization model with a simple but effective mechanism to enable users to specify high level attributes in order to control the shape of the final summaries to better suit their needs.
Frames: a corpus for adding memory to goal-oriented dialogue systems
221 Citations2017Layla El Asri, Hannes Schulz +6 more
A rule-based baseline is proposed and the frame tracking task is proposed, which consists of keeping track of different semantic frames throughout each dialogue, and the task is analysed through this baseline.
Controlling Output Length in Neural Encoder-Decoders
206 Citations2016Yuta Kikuchi, Graham Neubig +3 more
Two decoding-based and two learning-based methods are proposed for controlling the output sequence length for neural encoder-decoder models and show that they have the capability to control length without degrading summary quality in a summarization task.
Hafez: an Interactive Poetry Generation System
135 Citations2017Marjan Ghazvininejad, Xing Shi +2 more
Hafez is an automatic poetry generation system that integrates a Recurrent Neural Network (RNN) with a Finite State Accep-tor (FSA) and enables users to revise and polish generated poems by adjusting various style configurations.
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
93 Citations2018Ashutosh Baheti, Alan Ritter +2 more
This work proposes a simple yet effective approach for incorporating side information in the form of distributional constraints over the generated responses that generates responses that are much less generic without sacrificing plausibility.
Proceedings of the AAAI Conference on Artificial IntelligenceMechanism-Aware Neural Machine for Dialogue Response Generation
90 Citations2017Ganbin Zhou, Ping Luo +4 more
This study proposes a mechanism-aware neural machine for dialogue response generation that assumes that there exists some latent responding mechanisms, each of which can generate different responses for a single input post, and develops a encoder-diverter-decoder framework to train its modules in an end-to-end fashion.
Learning to Control the Specificity in Neural Response Generation
86 Citations2018Ruqing Zhang, Jiafeng Guo +4 more
A novel controlled response generation mechanism to handle different utterance-response relationships in terms of specificity is proposed, which introduces an explicit specificity control variable into a sequence-to-sequence model, which interacts with the usage representation of words through a Gaussian Kernel layer.
Steering Output Style and Topic in Neural Response Generation
66 Citations2017Di Wang, Nebojša Jojić +2 more
The decompose the neural generation process into empirically easier sub-problems: a faithfulness model and a decoding method based on selective-sampling and training and sampling algorithms that bias the generation process with a specific language style restriction, or a topic restriction.
arXiv (Cornell University)Learning to Decode for Future Success
50 Citations2017Jiwei Li, Will Monroe +1 more
A simple, general strategy to manipulate the behavior of a neural decoder that enables it to generate outputs that have specific properties of interest, such as sequence length and backward probability, which is able to incorporate a variety of properties that cannot be handled by standard neural sequence decoders.
arXiv (Cornell University)Importance of a Search Strategy in Neural Dialogue Modelling
42 Citations2018Ilya Kulikov, Alexander Miller +2 more
A significant gap is observed between greedy search and the proposed iterative beam search augmented with selection scoring, demonstrating the importance of the search algorithm in neural dialogue generation.
arXiv (Cornell University)Topic-based Evaluation for Conversational Bots
41 Citations2018Fenfei Guo, Angeliki Metallinou +4 more
This work proposes to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle.
Metrics and Evaluation of Spoken Dialogue Systems
23 Citations2012Helen Hastie
This chapter examines metrics for evaluating system performance and dialogue quality and discusses evaluation techniques that can be used to automatically detect problems in the dialogue, thus filtering out good dialogues and leaving poor dialogues for further evaluation and investigation.
arXiv (Cornell University)Extending Neural Generative Conversational Model using External Knowledge Sources
7 Citations2018Prasanna Parthasarathi, Joëlle Pineau