Automatic detection of poor speech recognition at the dialogue level
Published 1 January 1999Open access
Diane Litman, Marilyn Walker, Michael Kearns
Citations64
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper adopts a machine learning approach to learn rules from a dialogue corpus for identifying situations where the speech recognizer is performing poorly and shows a significant improvement over the baseline.
Abstract
The dialogue strategies used by a spoken dialogue system strongly influence performance and user satisfaction. An ideal system would not use a single fixed strategy, but would adapt to the circumstances at hand. To do so, a system must be able to identify dialogue properties that suggest adaptation. This paper focuses on identifying situations where the speech recognizer is performing poorly. We adopt a machine learning approach to learn rules from a dialogue corpus for identifying these situations. Our results show a significant improvement over the baseline and illustrate that both lower-level acoustic features and higher-level dialogue features can affect the performance of the learning algorithm.
Keywords
Computer Science
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
Machine LearningBoosTexter: A Boosting-based System for Text Categorization
2,194 Citations2000Robert E. Schapire, Yoram Singer
This work describes in detail an implementation, called BoosTexter, of the new boosting algorithms for text categorization tasks, and presents results comparing the performance of Boos Texter and a number of other text-categorization algorithms on a variety of tasks.
Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems
906 Citations1991Sholom M. Weiss, Casimir A. Kulikowski
This book discusses how to Estimate the True Performance of a Learning System, and the Importance of Unbiased Error Rate Estimation, and Machine Learning: Easily Understood Decision Rules.
Choice Reviews OnlineEmpirical methods for artificial intelligence
637 Citations1996
Empirical research exploratory data analysis basic issues in experiment design hypothesis testing and estimation computer-intensive statistical methods performance assessment explaining performance - interactions and dependencies modelling tactics for generalization.
Learning trees and rules with set-valued features
244 Citations1996William W. Cohen
It is argued that many decision tree and rule learning algorithms can be easily extended to set-valued features, and it is shown by example that many real-world learning problems can be efficiently and naturally represented with set- valued features.
Computer Speech & LanguageEvaluating spoken dialogue agents with PARADISE: Two case studies
152 Citations1998Walker Ma, DJ Litman +2 more
PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating and comparing the performance of spoken dialogue agents, is presented and can be used both for making predictions about future versions of an agent, and as feedback to the agent so that the agent can learn to optimize its behaviour based on its experiences with users over time.
Learning optimal dialogue strategies
127 Citations1998Marilyn Walker, Jeanne C. Fromer +1 more
A novel method by which a dialogue agent can learn to choose an optimal dialogue strategy is described, based on a combination of learning algorithms and empirical evaluation techniques.
Characterizing and recognizing spoken corrections in human-computer dialogue
83 Citations1998Gina‐Anne Levow
Interestingly, corrections of misrecognition erros (CME) exhibited significantly heightened pitch variability, while corrections of rejection errors (CRE) showed only a small but significant decrease in pitch minimum, which allows the development of decision trees which distinguish CME's from CRE's and from original inputs at 70--75% accuracy based on duration, pitch, and amplitude features.
What can I say?
73 Citations1998Marilyn Walker, Jeanne C. Fromer +3 more
This paper presents experimental results comparing two different designs for a spoken language interface to entail, and shows that even though the mixedinitiative system is more eflicient, as measured by number of turns, or elapsed time to complete a set of email tasks, users prefer the system- initiative interface.
Human-machine problem solving using spoken language systems (SLS)
67 Citations1992Elizabeth Shriberg, E. C. S. Wade +1 more
While users may adapt to some aspects of an SLS, certain types of user behavior may require technological solutions and hyperarticulation increases recognition errors, and while instructions can reduce this behavior, they do not result in improved recognition performance.
Courses and lecturesEmpirically Evaluating an Adaptable Spoken Dialogue System
67 Citations1999Diane Litman, Shimei Pan
An empirical evaluation of TOOT, an adaptable spoken dialogue system for retrieving train schedules on the web, shows that adaptable TOOT generally outperforms non-adaptable ToOT, and that the utility of adaptation depends on TOOT’s initial dialogue strategies.
From novice to expert: the effect of tutorials on user expertise with spoken dialogue systems
59 Citations1998Candace Kamm, Diane Litman +1 more
This study explored the use of a 4-minute tutorial session to acquaint novice users with the features of a spoken dialogue system for accessing email and found that perceived task completion, mean recognition score, and number of help requests were significant predictors of user satisfaction with the system.
Evaluating response strategies in a Web-based spoken dialogue agent
49 Citations1998Diane Litman, Shimei Pan +1 more
An experimental evaluation of two alternative response strategies in TOOT, a spoken dialogue agent that allows users to access train schedules stored on the web via a telephone conversation, and identifies interdependencies that exist between speech recognition and response strategy.
arXiv (Cornell University)Evaluating Competing Agent Strategies for a Voice Email Agent
31 Citations1997Marilyn Walker, Donald Hindle +3 more
International Journal of Human-Computer StudiesAn evaluation of strategies for selectively verifying utterance meanings in spoken natural language dialog
26 Citations1998Ronnie W. Smith
This paper reports on both context-independent and context-dependent strategies for utterance verification that show that the use of dialog context can be very helpful in selecting which utterances to verify.
Evaluating competing agent strategies for a voice email agent
25 Citations1997Marilyn Walker, Donald Hindle +3 more
This paper reports experimental results comparing a mixed-initiative to a system- initiative dialog strategy in the context of a personal voice email agent and tests the PARADISE evaluation framework, and discusses the performance function derivable from the experimental data.
Proceedings of the 17th international conference on Computational linguistics -Characterizing and recognizing spoken corrections in human-computer dialogue
13 Citations1998Gina‐Anne Levow
Decoding optimal state sequence with smooth state likelihoods
11 Citations2002Ilija Zeljković
This paper presents a method for decoding state sequences with less volatile state probabilities by introducing penalties proportional to the difference of the current state likelihood and the highest state likelihood for the particular time frame in the Viterbi forward path at every time frame.
Adaptive mixed-initiative dialogue management
9 Citations2002Gert Veldhuijzen van Zanten
An architecture that supports adaptive mixed-initiative dialogue is described that is based on a generalisation of the form-filling paradigm that contains slots at various levels of abstraction and a question hierarchy is defined that allows for adaptive mixing of initiative in dialogue.
Proceedings of the 17th international conference on Computational linguistics -Evaluating response strategies in a Web-based spoken dialogue agent
9 Citations1998Diane Litman, Shimei Pan +1 more