Emotion Recognition Based on Joint Visual and Audio Cues
Published 1 January 2006
Nicu Sebe, Ira L. Cohen, Theo Gevers, Thomas S. Huang
Citations203
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The problem of bimodal emotion recognition is described and the use of probabilistic graphical models when fusing the different modalities is advocated.
Abstract
Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. However, one necessary ingredient for natural interaction is still missing - emotions. This paper describes the problem of bimodal emotion recognition and advocates the use of probabilistic graphical models when fusing the different modalities. We test our audio-visual emotion recognition approach on 38 subjects with 11 HCI-related affect states. The experimental results show that the average person-dependent emotion recognition accuracy is greatly improved when both visual and audio information are used in classification.
Keywords
PsychologyComputer Science
Computer Vision and Image UnderstandingFacial expression recognition from video sequences: temporal and static modeling
873 Citations2003Ira L. Cohen, Nicu Sebe +3 more
This work introduces and test different Bayesian network classifiers for classifying expressions from video, focusing on changes in distribution assumptions, and feature dependency structures, and proposes a new architecture of hidden Markov models (HMMs) for automatically segmenting and recognizing human facial expression from video sequences.
Proceedings of the IEEEToward an affect-sensitive multimodal human-computer interaction
784 Citations2003Maja Pantić, Léon Rothkrantz
It is argued that next-generation human-computer interaction (HCI) designs need to include the essence of emotional intelligence - the ability to recognize a user's affective states- in order to become more human-like, more effective, and more efficient.
Affective multimodal human-computer interaction
219 Citations2005Maja Pantić, Nicu Sebe +2 more
The components of human affect, how they might be integrated into computers, and how far are the authors from realizing affective multimodal human-computer interaction are discussed.
Bimodal emotion recognition
198 Citations2002L.C. De Silva, Pei Chi Ng
The findings show that the audio and video information can be combined using a rule-based system to improve the recognition rate.
WORLD SCIENTIFIC eBooksMULTIMODAL EMOTION RECOGNITION
191 Citations2005Nicu Sebe, Ira L. Cohen +1 more
New ways of human-computer interaction that enable the computer to be more aware of the user’s emotional and attentional expressions are explored.
Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face
140 Citations2002Yasunari Yoshitomi, Sung-Ill Kim +2 more
Audio-visual based emotion recognition-a new approach
126 Citations2004Mingli Song, Jiajun Bu +2 more
A novel approach is presented in this paper, including both visual and audio from video clips, to recognize the human emotion, and shows that this approach outperforms only using visual or audio separately.
User Modeling and User-Adapted InteractionModeling Multimodal Expression of User’s Affective Subjective Experience
95 Citations2002Nadia Bianchi‐Berthouze, Christine Lisetti
MOUE is an emotion recognition system that recognizes the user’s emotion from his/her facial expressions, and from it, adaptively builds semantic definitions of emotion concepts using the user's feedback; MIKE is an interactive adaptive system that, along with the user, co-evolves a language for communicating over subjective impressions.
Emotional expressions in audiovisual human computer interaction
83 Citations2002Li Chen, Thomas S. Huang
A system to continuously monitor the user's voice and facial motions for recognizing emotional expressions is described, crucial for intelligent computers that take on a social role such as an actor or a companion.
Connected vibrations: a modal analysis approach for non-rigid motion tracking
75 Citations2002Tao Hai, Thomas S. Huang
A new framework for tracking non-rigid motion in image sequences as the deformations of connected 2D membrane patches using stochastic filtering technique to improve the robustness of the tracker and promising results on both natural and synthetic facial motion sequences are demonstrated.
Bimodal HCI-related affect recognition
61 Citations2004Zhihong Zeng, Jilin Tu +7 more
This paper applies a smoothing method to extract the information of the affective state from facial features and proposes a bimodal affect recognition approach that takes into account some special affective states which indicate users' cognitive/motivational states.
Proceedings of the IEEEBoosted learning in dynamic bayesian networks for multimodal speaker detection
51 Citations2003A. Garg, Vladimir Pavlović +1 more
A supervised learning framework for Bayesian networks, which is based on the Adaboost algorithm of Schapire and Freund, is developed and tested in the context of a novel multimodal HCI application: a speech-based command and control interface for a Smart Kiosk.