Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
Journal of Artificial Intelligence ResearchPublished 1 February 2002Open access
Satinder Singh, Diane Litman, Michael Kearns, Marilyn Walker
Citations353
SJR quartileQ1
SJR score1.37
SNIP2.97
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The design, construction and empirical evaluation of NJFun, an experimental spoken dialogue system that provides users with access to information about fun things to do in New Jersey, are reported on.
Abstract
Designing the dialogue policy of a spoken dialogue system involves many nontrivial choices. This paper presents a reinforcement learning approach for automatically optimizing a dialogue policy, which addresses the technical challenges in applying reinforcement learning to a working dialogue system with human users. We report on the design, construction and empirical evaluation of NJFun, an experimental spoken dialogue system that provides users with access to information about fun things to do in New Jersey. Our results show that by optimizing its performance via reinforcement learning, NJFun measurably improves system performance.
Keywords
Computer Science
Journal of Artificial Intelligence ResearchReinforcement Learning: A Survey
8,831 Citations1996Leslie Pack Kaelbling, Michael L. Littman +1 more
Central issues of reinforcement learning are discussed, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state.
MIT Press eBooksIntroduction to Reinforcement Learning
6,868 Citations1998Richard S. Sutton, Andrew G. Barto
In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning.
IEEE Transactions on Speech and Audio ProcessingA stochastic model of human-machine interaction for learning dialog strategies
523 Citations2000Esther Levin, Roberto Pieraccini +1 more
The experimental results show that it is indeed possible to find a simple criterion, a state space representation, and a simulated user parameterization in order to automatically learn a relatively complex dialog behavior, similar to one that was heuristically designed by several research groups.
Improving Elevator Performance Using Reinforcement Learning
495 Citations1995Robert H. Crites, Andrew G. Barto
Results in simulation surpass the best of the heuristic elevator control algorithms of which the author is aware and demonstrate the power of RL on a very large scale stochastic dynamic optimization problem of practical utility.
Mixed initiative in dialogue
219 Citations1990Marilyn Walker, Steve Whittaker
A set of rules for the transfer of control is applied to 4 sets of dialogues and it is found that both allocation of control and the manner in which control is transferred is radically different for the two dialogue types.
Reinforcement Learning for Spoken Dialogue Systems
206 Citations1999Satinder Singh, Michael Kearns +2 more
A general software tool (RLDS, for Reinforcement Learning for Dialogue Systems) based on the MDP framework is built and applied to dialogue corpora gathered from two dialogue systems built at AT&T Labs, demonstrating that RLDS holds promise as a tool for "browsing" and understanding correlations in complex, temporally dependent dialogue Corpora.
An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email
185 Citations2000Marilyn Walker Walker
ArXiv.orgMetrics for Evaluating Dialogue Strategies in a Spoken Language System
155 Citations1996Morena Danieli, Elisabetta Gerbino
A new metric is defined, the implicit recovery, which allows to measure the ability of a dialogue manager to deal with errors by different levels of analysis and is compared to two different approaches to dialogue repair strategies.
Computer Speech & LanguageEvaluating spoken dialogue agents with PARADISE: Two case studies
152 Citations1998Walker Ma, DJ Litman +2 more
PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating and comparing the performance of spoken dialogue agents, is presented and can be used both for making predictions about future versions of an agent, and as feedback to the agent so that the agent can learn to optimize its behaviour based on its experiences with users over time.
Learning optimal dialogue strategies
127 Citations1998Marilyn Walker, Jeanne C. Fromer +1 more
A novel method by which a dialogue agent can learn to choose an optimal dialogue strategy is described, based on a combination of learning algorithms and empirical evaluation techniques.
SPoT
109 Citations2001Marilyn Walker, Owen Rambow +1 more
This paper presents SPoT, a sentence planner, and a new methodology for automatically trainingSPoT on the basis of feedback provided by human judges, and shows that the trained SPR learns to select a sentence plan whose rating on average is only 5% worse than the top human-ranked sentence plan.
Automatic detection of poor speech recognition at the dialogue level
64 Citations1999Diane Litman, Marilyn Walker +1 more
This paper adopts a machine learning approach to learn rules from a dialogue corpus for identifying situations where the speech recognizer is performing poorly and shows a significant improvement over the baseline.
From novice to expert: the effect of tutorials on user expertise with spoken dialogue systems
59 Citations1998Candace Kamm, Diane Litman +1 more
This study explored the use of a 4-minute tutorial session to acquaint novice users with the features of a spoken dialogue system for accessing email and found that perceived task completion, mean recognition score, and number of help requests were significant predictors of user satisfaction with the system.
Evaluating response strategies in a Web-based spoken dialogue agent
49 Citations1998Diane Litman, Shimei Pan +1 more
An experimental evaluation of two alternative response strategies in TOOT, a spoken dialogue agent that allows users to access train schedules stored on the web via a telephone conversation, and identifies interdependencies that exist between speech recognition and response strategy.
Predicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System
44 Citations2000Diane Litman, Shimei Pan
The design and evaluation of an adaptive version of TOOT, a spoken dialogue system for retrieving online train schedules that predicts whether a user is having speech recognition problems as a particular dialogue progresses, and automatically adapts its dialogue strategies based on its predictions.
International Journal of Human-Computer StudiesAn evaluation of strategies for selectively verifying utterance meanings in spoken natural language dialog
26 Citations1998Ronnie W. Smith
This paper reports on both context-independent and context-dependent strategies for utterance verification that show that the use of dialog context can be very helpful in selecting which utterances to verify.
Evaluation of the Dutch train timetable information system developed in the ARISE project
23 Citations2002Angelien A. Sanderman, J. Sturm +3 more
An evaluation of a version of a train time table information system that combines explicit verification with mixed-initiative dialogue control in the first part of the interaction found subjects found it difficult to deal with the open questions in the second part of an interaction, that were meant to facilitate navigation.
A dialog control strategy based on the reliability of speech recognition
15 Citations2002Yasuhisa Niimi, Yutaka Kobayashi
The analysis has proven that the direct confirmation can increase P/sub ac/ and the indirect confirmation can reduce N in comparison with the basic strategy, and a numerical evaluation of the dialog control strategy conducted by using a real speech recognition system.
ArXiv.orgTracking Initiative in Collaborative Dialogue Interactions
11 Citations1997Jennifer Chu‐Carroll, Michael K. Brown