Dynamic Facial Emotion Recognition Oriented to HCI Applications
Interacting with ComputersPublished 14 November 2013Open access
Samuel Marcos-Pablos, J. G. Garcia-Bermejo, E. ZalamaCasanova, José Antonio Juárez López
Citations16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Experimental results carried out on live video sequences and on the Cohn-Kanade face database show that the proposed method provides high recognition hit rates.
Abstract
As part of a multimodal animated avatar previously presented in Marcos-Pablos et al. ((2010) A realistic, virtual head for human-computer interaction. Interact. Comput., 22, 176–192, ISSN 0953-5438), in this paper we describe a method for dynamic recognition of displayed facial emotions on low-resolution streaming images. First, we address the detection of action units (AUs) of the facial action coding system using active shape models and Gabor filters. Normalized outputs of the AU recognition step are then used as inputs for a neural network that consists of an habituation network plus a competitive network. Both the competitive and the habituation layer use differential equations, thus taking into account the dynamic information of facial expressions through time. Experimental results carried out on live video sequences and on the Cohn-Kanade face database show that the proposed method provides high recognition hit rates. To assess the suitability of the developed emotional recognition system for human–computer interaction applications, it has been successfully integrated in the architecture of an avatar and we have conducted a preliminary experiment on empathy. The experiment showed promising results, as the avatar that made use of the emotional recognition system obtained a clear increase in the positivity of the rating when compared with the same avatar with no emotional response.
Keywords
PsychologyComputer Science
The Journal of PhysiologyA quantitative description of membrane current and its application to conduction and excitation in nerve
23,234 Citations1952A. L. Hodgkin, A. F. Huxley
This article concludes a series of papers concerned with the flow of electric current through the surface membrane of a giant nerve fibre by putting them into mathematical form and showing that they will account for conduction and excitation in quantitative terms.
International Journal of Computer VisionRobust Real-Time Face Detection
14,193 Citations2004Paul Viola, Michael Jones
A new image representation called the “Integral Image” is introduced which allows the features used by the detector to be computed very quickly and a method for combining classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising face-like regions.
Computer Vision and Image UnderstandingActive Shape Models-Their Training and Application
7,187 Citations1995T.F. Cootes, Chris Taylor +2 more
This work describes a method for building models by learning patterns of variability from a training set of correctly annotated images that can be used for image search in an iterative refinement algorithm analogous to that employed by Active Contour Models (Snakes).
Journal of the Optical Society of America AUncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters
3,165 Citations1985John Daugman
Evidence is presented that the 2D receptive-field profiles of simple cells in mammalian visual cortex are well described by members of this optimal 2D filter family, and thus such visual neurons could be said to optimize the general uncertainty relations for joint 2D-spatial-2D-spectral information resolution.
Comprehensive database for facial expression analysis
2,582 Citations2002Takeo Kanade, Jeffrey F. Cohn +1 more
The problem space for facial expression analysis is described, which includes level of description, transitions among expressions, eliciting conditions, reliability and validity of training and test data, individual differences in subjects, head orientation and scene complexity image characteristics, and relation to non-verbal behavior.
Image and Vision ComputingFacial expression recognition based on Local Binary Patterns: A comprehensive study
2,259 Citations2008Caifeng Shan, Shaogang Gong +1 more
This paper empirically evaluates facial representation based on statistical local features, Local Binary Patterns, for person-independent facial expression recognition, and observes that LBP features perform stably and robustly over a useful range of low resolutions of face images, and yield promising performance in compressed low-resolution video sequences captured in real-world environments.
IEEE Transactions on Pattern Analysis and Machine IntelligenceRecognizing action units for facial expression analysis
1,741 Citations2001Yingli Tian, Takeo Kanade +1 more
An Automatic Face Analysis (AFA) system to analyze facial expressions based on both permanent facial features and transient facial features in a nearly frontal-view face image sequence and Multistate face and facial component models are proposed for tracking and modeling the various facial features.
Pattern RecognitionA survey of skin-color modeling and detection methods
926 Citations2006P. Kakumanu, Sokratis Makrogiannis +1 more
A critical up-to-date review of the various skin modeling and classification strategies based on color information in the visual spectrum and presents various approaches that use skin-color constancy and dynamic adaptation techniques to improve the skin detection performance in dynamically changing illumination and environmental conditions.
Boston studies in the philosophy of scienceContour Enhancement, Short Term Memory, and Constancies in Reverberating Neural Networks
650 Citations1982Stephen Grossberg
It is suggested that competition solves a sensitivity problem that confronts all cellular systems: the noise-saturation dilemma.
Lecture notes in computer scienceLocating Facial Features with an Extended Active Shape Model
629 Citations2008Stephen Milborrow, Fred Nicolls
This work makes some simple extensions to the Active Shape Model, and uses it to locate features in frontal views of upright faces and shows on independent test data that with the extensions the Active Shape Model compares favorably with more sophisticated methods.
IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences
580 Citations2006Maja Pantić, Ioannis Patras
This paper presents a system for automatic recognition of facial action units (AUs) and their temporal models from long, profile-view face image sequences and introduces facial-action-dynamics recognition from continuous video input using temporal rules.
International Journal of Human-Computer StudiesComputers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent
508 Citations2005Scott Brave, Clifford Nass +1 more
Emppathic emotion was found to have a significant effect on the agent's likability, trustworthiness, and perceived caring, as well as on the user's felt support, independent of the presence of self-oriented emotion.
Fully Automatic Facial Action Unit Detection and Temporal Analysis
351 Citations2006Michel Valstar, Maja Pantić
This work reports on the progress of building a system that enables fully automated fast and robust facial expression recognition from face video and analyses subtle changes in facial expression by recognizing facial muscle action units (AUs) and analysing their temporal behavior.
International Journal of Human-Computer StudiesAffective interaction: How emotional agents affect users
244 Citations2009Russell Beale, Chris Creed
A structured overview of the research into emotional simulation in agents is presented, providing a summary of the main studies, re-formulating appropriate results in terms of the emotional effects demonstrated, and an in-depth analysis illustrating the similarities and inconsistencies between different experiments across a variety of different domains.
Journal on Multimodal User InterfacesWhen my robot smiles at me: Enabling human-robot rapport via real-time head gesture mimicry
179 Citations2009Laurel D. Riek, Philip C. Paul +1 more
Haar Features for FACS AU Recognition
167 Citations2006Jacob Whitehill, Christian W. Omlin
Empirical results showed that the Haar+Adaboost method yields AU recognition rates comparable to those of the Gabor+SVM method but operates at least two orders of magnitude more quickly.
Image and Vision ComputingTrainable method of parametric shape description
165 Citations1992T.F. Cootes, D. H. Cooper +2 more
A trainable method of shape representation which can automatically capture the invariant properties of a class of shapes and provide a compact parametric description of variability is developed.
Proceedings of the National Academy of SciencesSOME NONLINEAR NETWORKS CAPABLE OF LEARNING A SPATIAL PATTERN OF ARBITRARY COMPLEXITY
148 Citations1968Stephen Grossberg
This note describes some nonlinear net networks, which learn a spatial pattern, in "black and white," of arbitrary size and complexity, and lists in heuristic terminology some of the properties which arise in the learning of patterns.
IEEE Intelligent SystemsGender-Specific Approaches to Developing Emotionally Intelligent Learning Companions
122 Citations2007Winslow Burleson, Rosalind W. Picard
According to research in this area, systems that provide affective support to frustrated users can reduce frustration.
Pattern RecognitionFacial expression recognition on multiple manifolds
97 Citations2010Rui Xiao, Qijun Zhao +2 more
A generalized framework for modeling and recognizing facial expressions on multiple manifolds is presented which assumes that different expressions may reside on different manifolds of possibly different dimensionalities.
Journal of Cross-Cultural PsychologyA Cross-Cultural Study of Recognition Thresholds for Facial Expressions of Emotion
84 Citations1986Francis T. McAndrew
Automatic Facial Expression Recognition Using Gabor Filter and Expression Analysis
77 Citations2010Jun Ou, Xiaobo Bai +3 more
A system that uses 28 facial feature key-points in images detection and Gabor wavelet filter provided with 5 frequencies, 8 orientations can extract the feature of low quality facial expression image target, and have good robust for automatic facial expression recognition.
Pattern Analysis and ApplicationsRecognising facial expressions in video sequences
60 Citations2007José M. Buenaposada, Enrique Muñoz +1 more
A model-based approach for facial expression recognition is adopted with a probabilistic procedure based on a nearest-neighbour approach to combine the information provided by the incoming image sequence with the prior information stored in the expression manifold to compute a posterior probability associated with a facial expression.
Neural Computing and ApplicationsFace recognition using Gabor-based complete Kernel Fisher Discriminant analysis with fractional power polynomial models
53 Citations2009Jun-Bao Li, Jeng‐Shyang Pan +1 more
A novel face recognition method by integrating the Gabor wavelet representation of face images and the enhanced powerful discriminator, complete Kernel Fisher Discriminant (CKFD) with fractional power polynomial (FPP) models is presented.
Facial Expression Recognition: A Fully Integrated Approach
44 Citations2007Roberto Valenti, Nicu Sebe +1 more
This paper presents a fully integrated system which performs facial features extraction and tracking accurately and in real time and represents a major step forward in the aim of achieving a humanlike interaction between the man and machine.
Interacting with ComputersA realistic, virtual head for human–computer interaction
44 Citations2009Samuel Marcos-Pablos, Jaime Gómez‐García‐Bermejo +1 more
An interactive and realistic virtual head oriented to human-computer interaction and social robotics is presented, designed following a hybrid approach, taking robotic characteristics into account and searching for a convergence between these characteristics, real facial actions and animation techniques.
Communicators' Perceptions of Social Presence as a Function of Avatar Realism in Small Display Mobile Communication Devices
41 Citations2008Sin‐Hwa Kang, James H. Watt +1 more
This study describes an experiment in which 126 participants engaged via a mobile telephone simulation that included a visual display in a discussion that required self-disclosure and affective evaluation of the other participant.
Persona effect revisited: Using bio-signals to measure and reflect the impact of character-based interfaces
40 Citations2003Helmut Prendinger, Sonja Mayer +2 more
Lecture notes in computer scienceA Hidden Markov Model Based Approach for Facial Expression Recognition in Image Sequences
25 Citations2010Miriam Schmidt, Martin Schels +1 more
The applicability of hidden Markov models for emotion recognition in image sequences is investigated, i.e. the temporal aspects of facial expressions, and two different fusion approaches were evaluated.
Lecture notes in computer scienceFacial Expression Recognition for Human-Robot Interaction – A Prototype
25 Citations2008Matthias Wimmer, Bruce A. MacDonald +2 more
This approach imposes lower computational requirements by specifically adapting model-based techniques to the FER scenario, and contains adaptive skin color extraction, localization of the entire face and facial components, and specifically learned objective functions for fitting a deformable face model.
2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition WorkshopsAutomatically detecting action units from faces of pain: Comparing shape and appearance features
24 Citations2009Patrick Lucey, Jeffrey F. Cohn +3 more
Using image data from patients with rotator-cuff injuries, this work describes an AAM-based automatic system that decouples shape and appearance to detect AUs on a frame-by-frame basis and found specific relationships between action units and types of facial features.
Categorical imperative<i>NOT</i>
24 Citations2004Diane J. Schiano, Sheryl M. Ehrlich +1 more
The research presented here systematically addresses the classic evidence for categorical perception of facial affect, using high-quality digital imaging and display technologies and improved research methods, and highlights the importance of preserving high-fidelity motion information in portraying emotion.
A realistic facial animation suitable for human-robot interfacing
19 Citations2008Samuel Marcos-Pablos, Jaime Gómez García Bermejo +1 more
An interactive and realistic facial animation oriented to human-computer interaction and social robotics is proposed, suitable for both real-time interfacing in human-machine interaction such as a social robot, and for helping in design and parametrization of android heads.
A model-based facial expression recognition algorithm using Principal Components Analysis
16 Citations2009Nicholas Vretos, Nikos Nikolaidis +1 more
The Candide facial grid is utilized and Principal Components Analysis (PCA) is applied to find the two eigenvectors of the model vertices to define a new coordinate system where vertices are mapped.
Pushy versus meek - using avatars to influence turn-taking behaviour
14 Citations2007Jens Edlund, Jonas Beskow
This article shows that a small set of simple interaction control gestures and a simple model of interaction can be used to influence users’ behaviour in an unobtrusive manner and implies that such a model may improve the flow of computer mediated interaction between humans under adverse circumstances, such as network latency.
Pattern Recognition LettersInterest filter vs. interest operator: Face recognition using Fisher linear discriminant based on interest filter representation
9 Citations2008Tuo Zhao, Zhizheng Liang +2 more
The interest Fisher classifier (IFC), which is robust to illumination and facial expression variability, applies the Fisher linear discriminant (FLD) to an augmented interest feature vector derived from interest filter representation of face images.
Nonverbal communication with a multimodal agent via facial expression recognition
8 Citations2011Samuel Marcos-Pablos, Jaime Gómez‐García‐Bermejo +2 more
The avatar's architecture and facial recognition system have been oriented to FACS Action Units recognition and generation, so they can be easily integrated and maintain nonverbal and verbal bidirectional communication.
Expression Recognition Based on VLBP and Optical Flow Mixed Features
5 Citations2009Jian Yi Kong, Yongzhao Zhan +1 more
This method uses fuzzy buried markov model to compute expression probability for each region, and applies the contribution weights obtained in the training stage to carry on the weighted-fusion and obtains the classification result.
A Fusion of Face Symmetry of Two-Dimensional Principal Component Analysis and Face Recognition
5 Citations2009Fulong Wang, Cheng Huang +1 more
A fusion of facial symmetry information method is developed for improving two-dimensional principal component analysis by weighting odd-even symmetry matrix to replace original image matrix extracting features, and at last minimum distance classifier is used for classification.
International Journal of Intelligent Systems Technologies and ApplicationsAutomated facial expression recognition &ndash; an integrated approach with optical flow analysis and Support Vector Machines
4 Citations2009Nanda Surendran, Sheng Quan Xie
In this article, an automated facial expression recognition system capable of classifying expressions by analysing the shape of the strongest facial feature responsible for the expression, namely the eyebrows or the mouth region is presented.
Lecture notes in computer scienceRecognizing Facial Expression Using Particle Filter Based Feature Points Tracker
4 Citations2007Rakesh Tripathi, R. Aravind
A particle filter based facial points tracker with a simple observation model based on sum-of-squared differences (SSD) between the intensities is developed and two kinds of classification schemes are considered, the hidden Markov model (HMM) as sequence based Recognizer and support vector machine (SVM) as frame based recognizer.