Affective multimodal human-computer interaction
Published 6 November 2005
Maja Pantić, Nicu Sebe, Jeffrey F. Cohn, Thomas S. Huang
Citations219
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The components of human affect, how they might be integrated into computers, and how far are the authors from realizing affective multimodal human-computer interaction are discussed.
Abstract
Social and emotional intelligence are aspects of human intelligence that have been argued to be better predictors than IQ for measuring aspects of success in life, especially in social interactions, learning, and adapting to what is important. When it comes to machines, not all of them will need such skills. Yet to have machines like computers, broadcast systems, and cars, capable of adapting to their users and of anticipating their wishes, endowing them with the ability to recognize user's affective states is necessary. This article discusses the components of human affect, how they might be integrated into computers, and how far are we from realizing affective multimodal human-computer interaction. Copyright © 2005 ACM.
Keywords
PsychologyNeuroscience
PsycEXTRA DatasetAffective computing
5,051 Citations1997Rosalind W. Picard
Key issues in affective computing, " computing that relates to, arises from, or influences emotions", are presented and new applications are presented for computer-assisted learning, perceptual information retrieval, arts and entertainment, and human health and interaction.
Handbook of Emotions
3,313 Citations1993Michael Lewis, Jeannette M. Haviland
Comprehensive database for facial expression analysis
2,582 Citations2002Takeo Kanade, Jeffrey F. Cohn +1 more
The problem space for facial expression analysis is described, which includes level of description, transitions among expressions, eliciting conditions, reliability and validity of training and test data, individual differences in subjects, head orientation and scene complexity image characteristics, and relation to non-verbal behavior.
IEEE Transactions on Pattern Analysis and Machine IntelligenceToward machine emotional intelligence: analysis of affective physiological state
2,302 Citations2001Rosalind W. Picard, Elias Vyzas +1 more
It is found that the technique of seeding a Fisher Projection with the results of sequential floating forward search improves the performance of the Fisher Projections and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral.
IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews)A Survey on Visual Surveillance of Object Motion and Behaviors
2,129 Citations2004Wenhan Hu, T.N. Tan +2 more
This paper reviews recent developments and general strategies of the processing framework of visual surveillance in dynamic scenes, and analyzes possible research directions, e.g., occlusion handling, a combination of two and three-dimensional tracking, and fusion of information from multiple sensors, and remote surveillance.
Psychological BulletinThin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.
1,841 Citations1992Nalini Ambady, Robert Rosenthal
Psychological BulletinIs there universal recognition of emotion from facial expression? A review of the cross-cultural studies.
1,614 Citations1994James A. Russell
Facial expressions and emotion labels are probably associated, but the association may vary with culture and is loose enough to be consistent with various alternative accounts, 8 of which are discussed.
Journal of Personality and Social PsychologyAcoustic profiles in vocal emotion expression.
1,534 Citations1996Rainer Banse, Klaus R. Scherer
Journal of Abnormal PsychologyTesting a tripartite model: I. Evaluating the convergent and discriminant validity of anxiety and depression symptom scales.
1,233 Citations1995David Watson, Kris Weber +3 more
Web-Based Database for Facial Expression Analysis
1,061 Citations2005Maja Pantić, Michel Valstar +2 more
The MMI facial expression database is presented, which includes more than 1500 samples of both static images and image sequences of faces in frontal and in profile view displaying various expressions of emotion, single and multiple facial muscle activation.
Speech CommunicationSpeech emotion recognition using hidden Markov models
907 Citations2003Tin Lay Nwe, Say Wei Foo +1 more
This paper proposes a text independent method of emotion classification of speech that makes use of short time log frequency power coefficients (LFPC) to represent the speech signals and a discrete hidden Markov model (HMM) as the classifier.
Computer Vision and Image UnderstandingFacial expression recognition from video sequences: temporal and static modeling
873 Citations2003Ira L. Cohen, Nicu Sebe +3 more
This work introduces and test different Bayesian network classifiers for classifying expressions from video, focusing on changes in distribution assumptions, and feature dependency structures, and proposes a new architecture of hidden Markov models (HMMs) for automatically segmenting and recognizing human facial expression from video sequences.
Journal of Abnormal PsychologyTesting a tripartite model: II. Exploring the symptom structure of anxiety and depression in student, adult, and patient samples.
861 Citations1995David Watson, Lee Anna Clark +3 more
Proceedings of the IEEEToward an affect-sensitive multimodal human-computer interaction
784 Citations2003Maja Pantić, Léon Rothkrantz
It is argued that next-generation human-computer interaction (HCI) designs need to include the essence of emotional intelligence - the ability to recognize a user's affective states- in order to become more human-like, more effective, and more efficient.
Motivation and EmotionCultural similarities and differences in display rules
624 Citations1990David Matsumoto
Facial Expression Of Emotion
510 Citations2002Dacher Keltner, Paul Ekman +2 more
Journal of Personality and Social PsychologyBehavioral markers and recognizability of the smile of enjoyment.
503 Citations1993Mark G. Frank, Paul Ekman +1 more
Results provide further evidence that enjoyment smiles are entities distinct from smiles in general.
interactionsHuman-centered design considered harmful
476 Citations2005Donald A. Norman
The purpose of this essay is to provoke thought, discussion, and reconsideration of some of the fundamental principles of human-centered design, which can be helpful, misleading, or wrong.
Oxford University Press eBooksBehavioral Markers and Recognizability of the Smile of Enjoyment
413 Citations2005Mark G. Frank, Paul Ekman +1 more
IEEE Transactions on Pattern Analysis and Machine IntelligenceLooking at people: sensing for ubiquitous and wearable computing
403 Citations2000Alex Pentland
The paper examines the mathematical tools that have proven successful, provides a taxonomy of the problem domain, and then examines the state of the art: person identification, surveillance/monitoring, 3D methods, and smart rooms/perceptual user interfaces.
Oxford University Press eBooksThe New Handbook of Methods in Nonverbal Behavior Research
359 Citations2008Jinni A. Harrigan, Robert Rosenthal +1 more
Vocal expression of affect
314 Citations2005Patrik N. Juslin, Klaus R. Scherer
International Journal of Wavelets Multiresolution and Information ProcessingTHE TIMING OF FACIAL MOTION IN POSED AND SPONTANEOUS SMILES
297 Citations2004Jeffrey F. Cohn, Karen L. Schmidt
These findings suggest that by extracting and representing dynamic as well as morphological features, automatic facial expression analysis can begin to discriminate among the message values of morphologically similar expressions.
ComputerSocially aware, computation and communication
252 Citations2005A. Pentlan
Technology must account for this by recognizing that communication is always socially situated and that discussions are not just words but part of a larger social dialogue, which forms a sort of collective intelligence.
Proceedings of the IEEEAudio-visual integration in multimodal communication
251 Citations1998Tsuhan Chen, R.R. Rao
This work reviews recent research that examines audio-visual integration in multimodal communication, including bimodality in human speech, human and automated lip reading, facial animation, lip synchronization, joint audio-video coding, andbimodal speaker verification.
Communications of the ACMHas the Internet become indispensable?
230 Citations2004Donna L. Hoffman, Thomas Novak +1 more
The idea is that the Internet has become so embedded in the daily fabric of people’s lives that they simply cannot live without it.
Bimodal emotion recognition
198 Citations2002L.C. De Silva, Pei Chi Ng
The findings show that the audio and video information can be combined using a rule-based system to improve the recognition rate.
Cambridge University Press eBooksThe new ethology of human facial expressions
191 Citations1997Alan J. Fridlund
Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face
140 Citations2002Yasunari Yoshitomi, Sung-Ill Kim +2 more
Audio-visual based emotion recognition-a new approach
126 Citations2004Mingli Song, Jiajun Bu +2 more
A novel approach is presented in this paper, including both visual and audio from video clips, to recognize the human emotion, and shows that this approach outperforms only using visual or audio separately.
Lecture notes in computer scienceEmotion Analysis in Man-Machine Interaction Systems
124 Citations2005Themis Balomenos, Amaryllis Raouzaiou +4 more
This paper presents a systematic approach to extracting expression related features from image sequences and inferring an emotional state via an intelligent rule-based system.
Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction
108 Citations2000Thomas S. Huang, Lawrence Shao-Hsien Chen
This thesis addresses the problem of detecting human emotional expressions by computer from the voice and facial motions of the user and proposes a complementary relationship between audio and video.
Embodied contextual agent in information delivering application
103 Citations2002Catherine Pélachaud, Valeria Carofiglio +3 more
The overall architecture of the conversational agent, a multimodal believable agent able to converse with the User by exhibiting a synchronized and coherent verbal and nonverbal behavior, and a dynamic model of emotions are described.
Authentic facial expression analysis
101 Citations2004Nicu Sebe, Michael S. Lew +4 more
Emotional expressions in audiovisual human computer interaction
83 Citations2002Li Chen, Thomas S. Huang
A system to continuously monitor the user's voice and facial motions for recognizing emotional expressions is described, crucial for intelligent computers that take on a social role such as an actor or a companion.
Proceedings of the IEEEUser-centered modeling and evaluation of multimodal interfaces
82 Citations2003Sharon Oviatt
This paper summarizes research on the cognitive science foundations of multimodal interaction, and on the essential role that user-centered modeling has played in prototyping, guiding, and evaluating the design of next-generation multi-modality interfaces, and describes the important role that selective methodologies and evaluation metrics have played in shaping next- generation multimodAL systems.
Emotion recognition from the facial image and speech signal
61 Citations2003Hyoun-Joo Go, Keun-Chang Kwak +2 more
An emotion recognition method using the facial images and speech signals and the LDA (linear discriminant analysis) results obtained better performance than previous ones.
Bimodal expression of emotion by face and voice
52 Citations1998Jeffrey F. Cohn, Gary S. Katz
The results of a series of studies developed semiautomated methods of discriminating emotion and para-linguistic communication in face and voice support the feasibility of human-computer interfaces that are sensitive to the full range of human nonverbal communication.
Proceedings of the IEEEBoosted learning in dynamic bayesian networks for multimodal speaker detection
51 Citations2003A. Garg, Vladimir Pavlović +1 more
A supervised learning framework for Bayesian networks, which is based on the Adaboost algorithm of Schapire and Freund, is developed and tested in the context of a novel multimodal HCI application: a speech-based command and control interface for a Smart Kiosk.
Audio-visual speaker detection using dynamic Bayesian networks
42 Citations2002A. Garg, Vladimir Pavlović +1 more
This work demonstrates the use of dynamic Bayesian networks (DBN) in tackling the problem of audio/visual speaker detection by optimally fused visual and audio sensors along with contextual information in a DBN architecture that infers instances when an individual is speaking.
Feature analysis for automatic speechreading
41 Citations2002Patricia Scanlon, Richard B. Reilly
It was observed that static features alone outperform a combination of both static and dynamic features when restricting the dimension of the feature vector, illustrating that the need for a certain level of detail in visual speech recognition is a higher priority than dynamic information.
Case-based reasoning for user-profiled recognition of emotions from face images
23 Citations2005Maja Pantić, Léon Rothkrantz
A case-based reasoning system capable of classifying facial expressions (given in terms of facial muscle actions) into the emotion categories learned from the user, based upon the user preferences and the generalizations formed from prior input is introduced.
Exploiting the dependencies in information fusion
21 Citations2003Hao Pan, Zhi‐Pei Liang +1 more
A formula is derived under the maximum entropy principle for estimating the joint probabilities of multisensory signals and selection of the mappings is guided by the maximum mutual information criterion.