Integrating information from speech and physiological signals to achieve emotional sensitivity
Published 4 September 2005
Jonghwa Kim, Elisabeth André, Matthias Rehm, Thurid Vogt, Johannes Wagner
Citations62
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Various methods for fusing physiological and voice data at the feature-level and the decision-level as well as a hybrid integration scheme and the results of the integrated recognition approach are compared with the individual recognition results from each modality.
Abstract
Recently, there has been a significant amount of work on the recognition of emotions from speech and biosignals. Most approaches to emotion recognition so far concentrate on a single modality and do not take advantage of the fact that an integrated multimodal analysis may help to resolve ambiguities and compensate for errors. In this paper, we describe various methods for fusing physiological and voice data at the feature-level and the decision-level as well as a hybrid integration scheme. The results of the integrated recognition approach are then compared with the individual recognition results from each modality.
Keywords
PsychologyComputer ScienceNeuroscience
IEEE Signal Processing MagazineEmotion recognition in human-computer interaction
2,507 Citations2001Roddy Cowie, Ellen Douglas‐Cowie +5 more
Basic issues in signal processing and analysis techniques for consolidating psychological and linguistic analyses of emotion are examined, motivated by the PKYSTA project, which aims to develop a hybrid system capable of using information from faces and voices to recognize people's emotions.
IEEE Transactions on Pattern Analysis and Machine IntelligenceToward machine emotional intelligence: analysis of affective physiological state
2,302 Citations2001Rosalind W. Picard, Elias Vyzas +1 more
It is found that the technique of seeding a Fisher Projection with the results of sequential floating forward search improves the performance of the Fisher Projections and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral.
American PsychologistThe emotion probe: Studies of motivation and attention.
2,267 Citations1995Peter J. Lang
Cognition & EmotionEmotion elicitation using films
2,182 Citations1995James J. Gross, Robert W. Levenson
Analysis of emotion recognition using facial expressions, speech and multimodal information
832 Citations2004Carlos Busso, Zhigang Deng +7 more
Results reveal that the system based on facial expression gave better performance than the systembased on just acoustic information for the emotions considered, and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.
From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification
512 Citations2005Johannes Wagner, Jonghwa Kim +1 more
This paper discussed the most important stages of a fully implemented emotion recognition system including data analysis and classification, and used a music induction method which elicits natural emotional reactions from the subject.
International Journal of Human-Computer StudiesThe production and recognition of emotions in speech: features and algorithms
438 Citations2003Oudeyer Pierre-Yves
A technique which allows to continuously control both the age of a synthetic voice and the quantity of emotions that are expressed and the first large-scale data mining experiment about the automatic recognition of basic emotions in informal everyday short utterances is presented.
Cognition Technology & WorkEmotion recognition from physiological signals using wireless sensors for presence technologies
365 Citations2004Fatma Nasoz, Kaye Alvarez +2 more
A new approach to enhance presence technologies that will adapt to user affect dynamically in the current context, thus providing enhanced social presence and introducing the prototype multimodal affective user interface.
Comparing Feature Sets for Acted and Spontaneous Speech in View of Automatic Emotion Recognition
224 Citations2005Thorwin Vogt, Elisabeth André
A data-mining experiment on feature selection for automatic emotion recognition using features derived from pitch, energy and MFCC time series finds the most relevant features in respect to the data are selected by removing correlated features.
Bimodal emotion recognition
198 Citations2002L.C. De Silva, Pei Chi Ng
The findings show that the audio and video information can be combined using a rule-based system to improve the recognition rate.
Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction
108 Citations2000Thomas S. Huang, Lawrence Shao-Hsien Chen
This thesis addresses the problem of detecting human emotional expressions by computer from the voice and facial motions of the user and proposes a complementary relationship between audio and video.
Emotional expressions in audiovisual human computer interaction
83 Citations2002Li Chen, Thomas S. Huang
A system to continuously monitor the user's voice and facial motions for recognizing emotional expressions is described, crucial for intelligent computers that take on a social role such as an actor or a companion.
Speech based emotion classification
73 Citations2002Tin Lay Nwe, Foo Say Wei +1 more
The recognizer presented in this paper is based on the discrete hidden Markov model and a novel feature vector based on mel frequency short time speech power coefficients is proposed.
Bimodal HCI-related affect recognition
61 Citations2004Zhihong Zeng, Jilin Tu +7 more
This paper applies a smoothing method to extract the information of the affective state from facial features and proposes a bimodal affect recognition approach that takes into account some special affective states which indicate users' cognitive/motivational states.
We are not amused - but how do you know? user states in a multi-modal dialogue system
34 Citations2003Anton Batliner, Viktor Zeißler +4 more
For the multi-modal dialogue system SmartKom, emotional user states in a Wizard-of-Oz experiment as, e.g., joyful, angry, helpless, are annotated holistically and based purely on facial expressions; other phenomena (prosodic peculiarities, offtalk, etc.) are labelled as well.