The CALO Meeting Assistant System
IEEE Transactions on Audio Speech and Language ProcessingPublished 16 February 2010Open access
Gökhan Tür, Andreas Stolcke, Lynn Voss, Stanley Peters, Dilek Hakkani‐Tür, John Dowding
Citations95
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The CALO-MA architecture and its speech recognition and understanding components, which include real-time and offline speech transcription, dialog act segmentation and tagging, topic identification and segmentation, question-answer pair identification, action item recognition, decision extraction, and summarization are presented.
Abstract
The CALO Meeting Assistant (MA) provides for distributed meeting capture, annotation, automatic transcription and semantic analysis of multiparty meetings, and is part of the larger CALO personal assistant system. This paper presents the CALO-MA architecture and its speech recognition and understanding components, which include real-time and offline speech transcription, dialog act segmentation and tagging, topic identification and segmentation, question-answer pair identification, action item recognition, decision extraction, and summarization.
Keywords
Computer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
ROUGE: A Package for Automatic Evaluation of Summaries
8,286 Citations2004Chin-Yew Lin
Four different RouGE measures are introduced: ROUGE-N, ROUge-L, R OUGE-W, and ROUAGE-S included in the Rouge summarization evaluation package and their evaluations.
ACM SIGIR ForumThe Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
2,210 Citations2017Jaime Carbonell, Jade Goldstein
A method for combining query-relevance with information-novelty in the context of text retrieval and summarization and preliminary results indicate some benefits for MMR diversity ranking in document retrieval and in single document summarization.
The use of MMR, diversity-based reranking for reordering documents and producing summaries
2,144 Citations1998Jaime Carbonell, Jade Goldstein
This paper presents a method for combining query-relevance with information-novelty in the context of text retrieval and summarization, and preliminary results indicate some benefits for MMR diversity ranking in document retrieval and in single document summarization.
TextTiling: segmenting text into multi-paragraph subtopic passages
1,253 Citations1997Marti A. Hearst
The algorithm is fully implemented and is shown to produce segmentation that corresponds well to human judgments of the subtopic boundaries of 12 texts, which should be useful for many text analysis tasks, including information retrieval and summarization.
Computational LinguisticsDialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
1,113 Citations2000Andreas Stolcke, Klaus Ries +8 more
A probabilistic integration of speech recognition with dialogue modeling is developed, to improve both speech recognition and dialogue act classification accuracy.
Machine LearningStatistical Models for Text Segmentation
687 Citations1999Doug Beeferman, Adam Berger +1 more
Assessment of the approach on quantitative and qualitative grounds demonstrates its effectiveness in two very different domains, Wall Street Journal news articles and television broadcast news story transcripts, using a new probabilistically motivated error metric.
Speech CommunicationProsody-based automatic segmentation of speech into sentences and topics
420 Citations2000Elizabeth Shriberg, Andreas Stolcke +2 more
This work combines prosodic cues with word-based approaches, and evaluates performance on two speech corpora, Broadcast News and Switchboard, finding that the prosodic model achieves comparable performance with significantly less training data, and requires no hand-labeling of prosodic events.
Discourse segmentation of multi-party conversation
351 Citations2003Michel Galley, Kathleen McKeown +2 more
A domain-independent topic segmentation algorithm for multi-party speech that combines knowledge about content using a text-based algorithm as a feature and about form using linguistic and acoustic cues about topic shifts extracted from speech.
The ICSI Meeting Recorder Dialog Act (MRDA) Corpus
349 Citations2004Elizabeth Shriberg, Raj Dhillon +3 more
A new corpus of over 180,000 hand- annotated dialog act tags and accompanying adjacency pair annotations for roughly 72 hours of speech from 75 naturally-occurring meetings is described.
Extractive summarization of meeting recordings
240 Citations2005Gabriel Murray, Steve Renals +1 more
This work contrasts feature-based approaches using prosodic and lexical features with maximal marginal relevance and latent semantic analysis approaches to summarization, and investigates how the summarization results might deteriorate when carried out on ASR output as opposed to manual transcripts.
Automatic Dialog Act Segmentation and Classification in Multiparty Meetings
180 Citations2006Jeremy Ang, Yang Liu +1 more
It is found that a very simple prosodic model aids performance over lexical information alone, especially for segmentation, in the two related tasks of dialog act segmentation and DA classification for speech from the ICSI Meeting Corpus.
Unsupervised topic modelling for multi-party spoken discourse
152 Citations2006Matthew Purver, Thomas L. Griffiths +2 more
It is shown how Bayesian inference in this generative model can be used to simultaneously address the problems of topic segmentation and topic identification: automatically segmenting multi-party meetings into topically coherent segments with performance which compares well with previous unsupervised segmentation-only methods.
Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization
150 Citations2005Sameer Maskey, Julia Hirschberg
It is shown that a summarization system that uses a combination of lexical, prosodic, structural and discourse features produces the most accurate summaries, and that a combinations of acoustic/prosodic and structural features are enough to build a ‘good’ summarizer when speech transcription is not available.
Statistical language modeling for speech disfluencies
138 Citations2002Andreas Stolcke, E. Shriberg
A language model is introduced that predicts disfluencies probabilistically and uses an edited, fluent context to predict following words and finds that the model reduces the word perplexity in the neighborhood of disfluency events; however, overall differences are small and have no significant impact on the recognition accuracy.
Computational Linguistics<b>Integrating Prosodic and Lexical Cues for Automatic Topic Segmentation</b>
127 Citations2001Gökhan Tür, Dilek Hakkani‐Tür +2 more
A probabilistic model that uses both prosodic and lexical cues for the automatic segmentation of speech into topically coherent units is presented and a significant reduction in error is achieved by combining the Prosodic and word-based knowledge sources.
The AMI System for the Transcription of Speech in Meetings
114 Citations2007Thomas Hain, Lukáš Burget +6 more
Using conditional random fields for sentence boundary detection in speech
107 Citations2005Yang Liu, Andreas Stolcke +2 more
The authors' CRF model yields a lower error rate than the HMM and Maxent models on the NIST sentence boundary detection task in speech, although it is interesting to note that the best results are achieved by three-way voting among the classifiers.
Meeting Browser: Tracking And Summarizing Meetings
101 Citations1998Alex Waibel, M. Bett +2 more
A probabilistic model based on context dependent phonetic rewrite rules to derive a list of possible pronunciations for all words or sequences of words is developed and developed to reduce the confusion of this expanded dictionary.
The ICSI Meeting Project: Resources and Research
96 Citations2004Adam Janin, J. Ang +10 more
A general description of the official ICSI Meeting Corpus is included, as currently available through the Linguistic Data Consortium, some of the existing and planned annotations which augment the basic transcripts provided there are discussed, and several research efforts that make use of these materials.
A global optimization framework for meeting summarization
91 Citations2009Dan Gillick, Korbinian Riedhammer +2 more
This work introduces a model for extractive meeting summarization based on the hypothesis that utterances convey bits of information, or concepts, and achieves ROUGE scores at least as good as a RouGE-based oracle derived from human summaries.
Task-Focused Summarization of Email
89 Citations2004Simon Corston-Oliver, Eric K. Ringger +2 more
SmartMail, a prototype system for automatically identifying action items (tasks) in email messages, presents the user with a task-focused summary of a message that contains a list of action items extracted from the message.
Statistical models for topic segmentation
89 Citations1999Jeffrey C. Reynar
New clues that mark shifts to new topics, novel algorithms for identifying topic boundaries and the uses of such boundaries once identified are described.
Addressee Identification In Face-to-Face Meetings
70 Citations2006Nedeljko Jovanovic, Rieks op den Akker +1 more
Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal ProcessingUsing corpus and knowledge-based similarity measure in Maximum Marginal Relevance for meeting summarization
66 Citations2008Shasha Xie, Yang Liu
This paper introduces a corpus- based measure to capture the similarity at the semantic level, and compares this method with cosine similarity and centroid score that only considers the salient words in the segments.
Dialog act classification with the help of prosody
63 Citations2002Marion Mast, Ralf Kompe +6 more
This paper presents automatic methods for the segmentation and classification of dialog acts (DA) in VERBMOBIL and uses polygrams and multi-layer perceptrons, using prosodic features.
Lecture notes in computer scienceThe Necessity of a Meeting Recording and Playback System, and the Benefit of Topic–Level Annotations to Meeting Browsing
61 Citations2005Satanjeev Banerjee, Carolyn Penstein Rosé +1 more
The proposed solution allows busy professionals to transcend time in a sense by revisiting a recorded meeting that has been structured for quick retrieval of sought information, and thus potentially facilitating the effective continuation of discussions from the past.
Lecture notes in computer scienceFurther Progress in Meeting Recognition: The ICSI-SRI Spring 2005 Speech-to-Text Evaluation System
61 Citations2006Andreas Stolcke, Xavier Anguera +8 more
This year's speech recognition system for the National Institute of Standards and Technology (NIST) Spring 2005 Meeting Rich Transcription (RT-05S) evaluation features better delay-sum processing of distant microphone channels and energy-based crosstalk suppression for close-talking microphones.
Text, speech and language technologyRecent Trends in Discourse and Dialogue
53 Citations2008Dybkjær, Laila, Wolfgang Minker +1 more
A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialogue Systems and Partially Observable Markov Decision Processes with Continuous Observations for Dialogue Management is presented.
Edinburgh Research Explorer (University of Edinburgh)Automatic Segmentation of Multiparty Dialogue
53 Citations2006Pei-Yun Hsueh, Johanna D. Moore +1 more
Examination of the effect of features shows that predicting top-level and predicting subtopic boundaries are two distinct tasks, and that the transcription errors inevitable in ASR output have a negative impact on models that combine lexical-cohesion and conversational features, but do not change the general preference of approach for the two tasks.
Detecting and Summarizing Action Items in Multi-Party Dialogue
52 Citations2007Matthew Purver, John Dowding +4 more
A semantic parser is described that identifies potential summarizing phrases, and it is shown that for some task properties these can be more informative than plain utterance transcriptions.
Lecture notes in computer scienceToward Joint Segmentation and Classification of Dialog Acts in Multiparty Meetings
45 Citations2006Matthias Zimmermann, Yang Liu +2 more
Packing the meeting summarization knapsack
45 Citations2008Korbinian Riedhammer, Dan Gillick +2 more
This work presents a method for determining the best possible extractive summary given an evaluation metric like ROUGE, and shows oracle and baseline results for a variety of evaluation scenarios that have recently appeared in this field.
Unsupervised Languagemodel Adaptation for Meeting Recognition
44 Citations2007Gökhan Tür, Andreas Stolcke
An application of unsupervised language model (ML) adaptation to meeting recognition, in a scenario where sequences of multiparty meetings on related topics are to be recognized, but no prior in-domain data for LM training is available.
Modelling and detecting decisions in multi-party dialogue
44 Citations2008Raquel Fernández, Matthew Frampton +3 more
A structured approach that takes into account the different roles utterances play in the decision-making process outperforms the accuracy achieved by existing decision detection systems based on flat annotations, while enabling the extraction of more fine-grained information that can be used for summarization and reporting.
The ICSI+ multilingual sentence segmentation system
43 Citations2006Molly E. Zimmerman, Dilek Hakkani‐Tür +5 more
It is demonstrated that the proposed methodology including pitch- and energyrelated prosodic features performs significantly better than a baseline system that uses words and simple pause features only.
The CALO meeting speech recognition and understanding system
42 Citations2008Gökhan Tür, Andreas Stolcke +19 more
The CALO-MA architecture and its speech recognition and understanding components, which include real-time and offline speech transcription, dialog act segmentation and tagging, question-answer pair identification, action item recognition, decision extraction, and summarization, are summarized.
A texttiling based approach to topic boundary detection in meetings
38 Citations2006Satanjeev Banerjee, Alexander I. Rudnicky
The goal is to automatically detect boundaries between discussions of different topics in meetings by adapting the TextTiling algorithm to the context of meetings, and the basic idea of the algorithm is sound, but breaks down when participants stray from typical behavior.
MODEL ADAPTATION FOR DIALOG ACT TAGGING
37 Citations2006Gökhan Tür, Ümit Güz +1 more
The results indicate that the tagger can achieve significantly better dialog act tagging by automatically selecting a subset of the Switchboard corpus and combining the confidences obtained by both in- domain and out-of-domain models via logistic regression, especially when the in-domain data is limited.
Detecting action-items in e-mail
37 Citations2005Paul N. Bennett, Jaime Carbonell
Co-adaptation: Adaptive co-training for semi-supervised learning
32 Citations2009Gökhan Tür
The goal is improving the performance of a dialog act segmentation model by exploiting the vast amount of unlabeled data by adapting the existing models using co-adaptation technique.
AUTOMATIC TOPIC SEGMENTATION AND LABELING IN MULTIPARTY DIALOGUE
32 Citations2006Pei-Yun Hsueh, Johanna D. Moore
Experiments in topic segmentation show that a supervised classification approach that combines lexical and conversational features outperforms the unsupervised lexical chain-based approach, achieving 20% and 12% improvement on segmentating top-level and sub-topic segments respectively.
ACM Transactions on Speech and Language ProcessingExtrinsic summarization evaluation
30 Citations2009Gabriel Murray, Thomas Kleinbauer +4 more
It is found that while ASR errors affect user satisfaction on an information retrieval task, users can adapt their browsing behavior to complete the task satisfactorily and consider extractive summaries to be intuitive and useful tools for browsing multimodal meeting data.
A keyphrase based approach to interactive meeting summarization
30 Citations2008Korbinian Riedhammer, Benoît Favre +1 more
This work introduces a simple yet robust algorithm to automatically extract keyphrases (KP) from a meeting which can then be used as a query in the MMR algorithm, and shows that the KP based system significantly outperforms both baseline and centroid based systems.
MODEL ADAPTATION FOR SENTENCE SEGMENTATION FROM SPEECH
26 Citations2006Sébastien Cuendet, Dilek Hakkani-Tür +1 more
The sentence segmentation performance is significantly improved by the adapted classification model compared to the one obtained by using in-domain data only, independently of the amount of in- domain data used.
Segmenting meetings into agenda items by extracting implicit supervision from human note-taking
25 Citations2007Satanjeev Banerjee, Alexander I. Rudnicky
This paper investigates the idea of acquiring implicit supervision from human meeting participants to solve the segmentation problem, and implements and tests a note taking interface that gives value to users by helping them organize and retrieve their notes easily, but that also extracts a segmentation of the meeting based on note taking behavior.
Co-training using prosodic and lexical information for sentence segmentation
24 Citations2007Ümit Güz, Sébastien Cuendet +2 more
This work investigates the application of the co-training learning algorithm on the sentence boundary classification problem by using lexical and prosodic information and the experimental results on the ICSI Meeting (MRDA) corpus show the effectiveness of this algorithm.
Identifying relevant phrases to summarize decisions in spoken meetings
23 Citations2008Raquel Fernández, Matthew Frampton +4 more
This work describes the approach to modelling decision discussions in spoken meetings and compares two approaches to extracting information from these discussions that uses an opendomain semantic parser and a machine learning technique.
Automatically detecting action items in audio meeting recordings
21 Citations2006William E. Morgan, Pi-Chuan Chang +2 more
A maximum entropy model is used to automatically detect action item-related utterances from multi-party audio meeting recordings and it is shown that on a corpus of action item annotations on the ICSI meeting recordings, the system performs at an F measure of 31.92%.
IEEE Transactions on Audio Speech and Language Processing</title> </titles> <publication_date> <month>09</month> <year>2008</year> </publication_date> <pages> <first_page>1359</first_page> <last_page>1359</last_page> </pages> <publisher_item> <item_number item_number_type='sequence-number'>4599404</item_number> </publisher_item> <doi_data> <doi>10.1109/TASL.2008.2004459</doi> <resource>http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4599404</resource> </doi_data> </journal_article> <journal_article> <titles> <title><![CDATA[A Probabilistic Model of Meetings That Combines Words and Discourse Features
19 Citations2008Mike Dowman, Virginia Savova +4 more
Lecture notes in computer scienceUsing Prosody for Automatic Sentence Segmentation of Multi-party Meetings
18 Citations2006Jáchym Kolář, Elizabeth Shriberg +1 more
Results show that information from pauses is important, including pause duration both at the boundary and at the previous and following word boundaries; and adding duration, pitch, and energy features yields significant improvement over pause alone.
Lecture notes in computer scienceExtrinsic Summarization Evaluation: A Decision Audit Task
18 Citations2008Gabriel Murray, Thomas Kleinbauer +4 more
University of Twente Research InformationFirst Steps Towards the Automatic Construction of Argument-Diagrams from Real Discussions
18 Citations2006Daan Verbree, Rutger Rienks +1 more
It is shown that unit labels of argument diagrams can be learnt and predicted by a computer with an accuracy of 78,52% and 51,43% on an unbalanced and balanced set respectively.
Word distributions for thematic segmentation in a support vector machine approach
16 Citations2006Maria Georgescul, Alexander Clark +1 more
Substantial improvement is obtained over previously published results of word-distribution based systems when evaluation is done on a corpus of recorded and transcribed multi-party dialogs.
Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal ProcessingExploiting dialogue act tagging and prosodic information for action item identification
15 Citations2008Fan Yang, Gökhan Tür +1 more
The results indicate that dialogue act tagging improves the identification of action item descriptions by 5% over lexical information, and prosodic information helps discriminating backchannels from agreements with 25% absolute improvement over a baseline.
Real-time decision detection in multi-party dialogue
12 Citations2009Matthew Frampton, Jia Huang +2 more
This paper describes a process for automatically detecting decision-making sub-dialogues in multi-party, human-human meetings in real-time, and shows that the resulting system's performance compares well with other detectors, including an off-line version.
Speaker adaptation of language models for automatic dialog act segmentation of meetings
11 Citations2007Jáchym Kolář, Yang Liu +1 more
This paper explores speaker adaptation of hidden event language models (LMs) for DA segmentation using the ICSI Meeting Corpus and finds the speaker-adapted LM yields statistically significant improvement over the baseline LM for both test conditions.
Lecture notes in computer scienceShallow Dialogue Processing Using Machine Learning Algorithms (or Not)
11 Citations2005Andréi Popescu-Belis, Alexander Clark +3 more
A rule-based approach is proposed for solving cross-modal references to meeting documents and the integration of the components into an automated shallow dialogue parser opens the way to multimodal meeting processing and retrieval applications.
Cascaded lexicalised classifiers for second-person reference resolution
11 Citations2009Matthew Purver, Raquel Fernández +2 more
This paper examines the resolution of the second person English pronoun you in multi-party dialogue, and shows that accuracy and robustness can be improved by use of simple lexical features, capturing the intuition that different uses and addressees are associated with different vocabularies.
North American Chapter of the Association for Computational LinguisticsCombining Probability-Based Rankers for Action-Item Detection
11 Citations2007Paul N. Bennett, Jaime Carbonell
This paper studies methods that automatically detect action-items in e-mail, an important category for assisting users in identifying new tasks, tracking ongoing ones, and searching for completed ones, to combine the rankings produced by different algorithms as well as different views.
Archive ouverte UNIGE (University of Geneva)Exploiting Structural Meeting-Specific Features for Topic Segmentation
9 Citations2007Maria Georgescul, Alexander Clarck +1 more
This investigation reports on the use of various lexical, acoustic and syntactic features, and makes a comparison of how these features influence performance of automatic topic segmentation.
Exploiting user feedback for language model adaptation in meeting recognition
7 Citations2009Dimitra Vergyri, Andreas Stolcke +1 more
It is found that recognition accuracy using the adapted LM can be enhanced substantially by partial correction, and if all content words are corrected, recognition improves to the same accuracy as if completely error-free transcriptions had been used for adaptation.