Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning
Published 1 January 2018Open access
Pararth Shah, Dilek Hakkani‐Tür, Bing Liu, Gökhan Tür
Citations127
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper discusses the advantages of this approach for industry applications of conversational agents, wherein an agent can be rapidly bootstrapped to deploy in front of users and further optimized via interactive learning from actual users of the system.
Abstract
Pararth Shah, Dilek Hakkani-Tür, Bing Liu, Gokhan Tür. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers). 2018.
Keywords
Computer Science
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
1,725 Citations2016Iulian Vlad Serban, Alessandro Sordoni +3 more
arXiv (Cornell University)A Neural Conversational Model
1,500 Citations2015Oriol Vinyals, Quoc V. Le
A simple approach to conversational modeling which uses the recently proposed sequence to sequence framework, and is able to extract knowledge from both a domain specific dataset, and from a large, noisy, and general domain dataset of movie subtitles.
Deep Reinforcement Learning for Dialogue Generation
1,058 Citations2016Jiwei Li, Will Monroe +4 more
This work simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity, non-repetitive turns, coherence, and ease of answering.
A Network-based End-to-End Trainable Task-oriented Dialogue System
806 Citations2017Tsung-Hsien Wen, David Vandyke +6 more
This work introduces a neural network-based text-in, text-out end-to-end trainable goal-oriented dialogue system along with a new way of collecting dialogue data based on a novel pipe-lined Wizard-of-Oz framework that can converse with human subjects naturally whilst helping them to accomplish tasks in a restaurant search domain.
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
702 Citations2017Iulian Vlad Serban, Alessandro Sordoni +5 more
Knowledge-Based SystemsWizard of Oz studies — why and how
696 Citations1993Nils Dahlbäck, Arne Jönsson +1 more
ACM Transactions on Information SystemsAn iterative design methodology for user-friendly natural language office information applications
676 Citations1984John F. Kelley
A six-step, iterative, empirical human factors design methodology was used to develop CAL, a natural language computer application to help computer-naive business professionals manage their personal calenders and its dictionaries.
Building a Semantic Parser Overnight
350 Citations2015Yushi Wang, Jonathan Berant +1 more
A new methodology is introduced that uses a simple grammar to generate logical forms paired with canonical utterances that are meant to cover the desired set of compositional operators and uses crowdsourcing to paraphrase these canonical utterance into natural utterances.
Agenda-based user simulation for bootstrapping a POMDP dialogue system
339 Citations2007Jost Schatzmann, Blaise Thomson +3 more
This paper investigates the problem of bootstrapping a statistical dialogue manager without access to training data and proposes a new probabilistic agenda-based method for simulating user behaviour and shows that the learned policy was highly competitive, with task completion rates above 90%.
The Dialog State Tracking Challenge
331 Citations2013J. D. Williams, Antoine Raux +2 more
The dialog state tracking challenge seeks to address this by providing a heterogeneous corpus of 15K human-computer dialogs in a standard format, along with a suite of 11 evaluation metrics, and shows that the suite of performance metrics cluster into 4 natural groups.
Frames: a corpus for adding memory to goal-oriented dialogue systems
221 Citations2017Layla El Asri, Hannes Schulz +6 more
A rule-based baseline is proposed and the frame tracking task is proposed, which consists of keeping track of different semantic frames throughout each dialogue, and the task is analysed through this baseline.
arXiv (Cornell University)Building a Conversational Agent Overnight with Dialogue Self-Play
161 Citations2018Pararth Shah, Dilek Hakkani‐Tür +5 more
A new corpus of 3,000 dialogues spanning 2 domains collected with M2M is proposed, and comparisons with popular dialogue datasets on the quality and diversity of the surface forms and dialogue flows are presented.
Dialogue & DiscourseTraining End-to-End Dialogue Systems with the Ubuntu Dialogue Corpus
140 Citations2017Ryan Lowe, Nissan Pow +4 more
This paper construct and train end-to-end neural network-based dialogue systems using an updated version of the recent Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words.
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management
118 Citations2017Pei-Hao Su, Paweł Budzianowski +3 more
A practical approach to learn deep RL-based dialogue policies and demonstrate their effectiveness in a task-oriented information seeking domain is demonstrated.
arXiv (Cornell University)Continuously Learning Neural Dialogue Management
105 Citations2016Pei-Hao Su, Milica Gašić +6 more
A unified neural network framework is proposed to enable the system to first learn by supervision from a set of dialogue data and then continuously improve its behaviour via reinforcement learning, all using gradient-based algorithms on one single model.
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
97 Citations2016Pei-Hao Su, Milica Gašić +6 more
An on-line learning framework whereby the dialogue policy is jointly trained alongside the reward model via active learning with a Gaussian process model is proposed.
2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)Iterative policy learning in end-to-end trainable task-oriented neural dialog models
94 Citations2017Bing Liu, Ian Lane
A deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents.
An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog
92 Citations2017Bing Liu, Ian Lane
This work presents a novel end-to-end trainable neural network model that is able to track dialog state, issue API calls to knowledge base (KB), and incorporate structured KB query results into system responses to successfully complete task-oriented dialogs.
arXiv (Cornell University)On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
81 Citations2016Pei-Hao Su, Milica Gašić +6 more
On-line policy optimisation of spoken dialogue systems via live interaction with human subjects
71 Citations2011Milica Gašić, Filip Jurčíček +3 more
An experiment to learn a policy for a real-world task directly from human interaction using rewards provided by users shows that a usable policy can be learnt in just a few hundred dialogues without needing a user simulator and using a learning strategy that reduces the risk of taking bad actions.
arXiv (Cornell University)Just ASK: Building an Architecture for Extensible Self-Service Spoken Language Understanding
56 Citations2017Anjishnu Kumar, Arpit Gupta +10 more
The design of the machine learning architecture that underlies the Alexa Skills Kit (ASK) is presented, which was the first Spoken Language Understanding Software Development Kit (SDK) for a virtual digital assistant, as far as the authors are aware.
arXiv (Cornell University)Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
52 Citations2015Pei-Hao Su, David Vandyke +5 more
arXiv (Cornell University)End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
51 Citations2017Bing Liu, Gökhan Tür +3 more
A neural network based task-oriented dialogue system that can be optimized end-to-end with deep reinforcement learning (RL) and shows that deep RL based optimization leads to significant improvement on task success rate and reduction in dialogue length comparing to supervised training model.
Learning from real users: rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems
25 Citations2015Pei-Hao Su, David Vandyke +5 more
Two neural network models that evaluate a sequence of turn-level features to rate the success of a dialogue and make no use of any prior knowledge of the user's task are presented.
Task Completion Transfer Learning for Reward Inference
23 Citations2016Layla El Asri, Romain Laroche +1 more
This paper introduces Task Completion Transfer Learning (TCTL): a method to exploit the exact knowledge of task completion on a corpus of dialogues scored by users in order to optimise online learning.
Task Completion Platform: A self-serve multi-domain goal oriented dialogue platform
21 Citations2016Paul Crook, Alex Marin +21 more
A multi-domain dialogue platform that can host and execute large numbers of goal-orientated dialogue tasks, and features a task configuration language, TaskForm, that allows the definition of each individual task to be decoupled from the overarching dialogue policy used by the platform to complete those tasks.