Multimodal human–computer interaction: A survey
Computer Vision and Image UnderstandingPublished 3 February 2007
Alejandro Jaimes, Nicu Sebe
Citations958
SJR quartileQ1
SJR score0.86
SNIP1.50
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
In this paper, we review the major approaches to multimodal human–computer interaction, giving an overview of the field from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for multimodal human–computer interaction (MMHCI) research.
Keywords
Computer Science
Fundamentals of speech recognition
7,687 Citations1993Lawrence R. Rabiner, Biing‐Hwang Juang
This book presents a meta-modelling framework for speech recognition that automates the very labor-intensive and therefore time-heavy and therefore expensive and expensive process of manually modeling speech.
ACM Computing SurveysFace recognition
6,167 Citations2003Wenyi Zhao, Rama Chellappa +2 more
This paper provides an up-to-date critical survey of still- and video-based face recognition research, and categorizes existing recognition techniques but also presents detailed descriptions of representative methods within each category.
Medical Entomology and ZoologyRules of Play: Game Design Fundamentals
5,328 Citations2003Katie Salen, Eric Zimmerman
This text offers an introduction to game design and a unified model for looking at all kinds of games, from board games and sports to computer and video games.
PsycEXTRA DatasetAffective computing
5,051 Citations1997Rosalind W. Picard
Key issues in affective computing, " computing that relates to, arises from, or influences emotions", are presented and new applications are presented for computer-assisted learning, perceptual information retrieval, arts and entertainment, and human health and interaction.
Human-Computer Interaction
4,812 Citations2013Elias G. Carayannis
Elsevier eBooksA FRAMEWORK FOR REPRESENTING KNOWLEDGE
4,545 Citations1988Marvin Minsky
The enormous problem of the volume of background common sense knowledge required to understand even very simple natural language texts is discussed and it is suggested that networks of frames are a reasonable approach to represent such knowledge.
The Psychology of Human-Computer Interaction
4,079 Citations2018Stuart K. Card, Thomas P. Moran +1 more
This work focuses on applying Psychology to Design Reprise, and the GOMS Model of Manuscript Editing, as an application to Information-Processing Psychology.
IEEE Transactions on Pattern Analysis and Machine IntelligenceDetecting faces in images: a survey
3,429 Citations2002Shuicheng Yan, David Kriegman +1 more
Robotics and Autonomous SystemsA survey of socially interactive robots
3,060 Citations2003Terrence Fong, Illah Nourbakhsh +1 more
The context for socially interactive robots is discussed, emphasizing the relationship to other research fields and the different forms of “social robots”, and a taxonomy of design methods and system components used to build socially interactive Robots is presented.
IEEE Transactions on Pattern Analysis and Machine IntelligenceThe recognition of human movement using temporal templates
2,784 Citations2001Aaron Bobick, James W. Davis
A view-based approach to the representation and recognition of human movement is presented, and a recognition method matching temporal templates against stored instances of views of known actions is developed.
Communications of the ACMSome computer science issues in ubiquitous computing
2,367 Citations1993Mark Weiser
What is new and different about the computer science in ubiquitous computing is explained, and a series of examples drawn from various subdisciplines of computer science are outlined.
IEEE Transactions on Pattern Analysis and Machine IntelligenceToward machine emotional intelligence: analysis of affective physiological state
2,302 Citations2001Rosalind W. Picard, Elias Vyzas +1 more
It is found that the technique of seeding a Fisher Projection with the results of sequential floating forward search improves the performance of the Fisher Projections and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral.
American PsychologistThe emotion probe: Studies of motivation and attention.
2,267 Citations1995Peter J. Lang
IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews)A Survey on Visual Surveillance of Object Motion and Behaviors
2,129 Citations2004Wenhan Hu, T.N. Tan +2 more
This paper reviews recent developments and general strategies of the processing framework of visual surveillance in dynamic scenes, and analyzes possible research directions, e.g., occlusion handling, a combination of two and three-dimensional tracking, and fusion of information from multiple sensors, and remote surveillance.
Psychological BulletinThin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.
1,841 Citations1992Nalini Ambady, Robert Rosenthal
Pattern RecognitionAutomatic facial expression analysis: a survey
1,790 Citations2002Beat Fasel, Juergen Luettin
This survey introduces the most prominent automatic facial expression analysis methods and systems presented in the literature and discusses issues such as face normalization, facial expression dynamics and facial expression intensity.
IEEE Transactions on Pattern Analysis and Machine IntelligenceVisual interpretation of hand gestures for human-computer interaction: a review
1,776 Citations1997Vladimir Pavlović, Rajeev Sharma +1 more
A fraction of the recycle slurry is treated with sulphuric acid to convert at least some of the gypsum to calcium sulphate hemihydrate and the slurry comprising hemihYDrate is returned to contact the mixture of phosphate rock, phosphoric acid and recycle Gypsum slurry.
Computer Vision and Image UnderstandingThe Visual Analysis of Human Movement: A Survey
1,767 Citations1999Dariu M. Gavrila
A number of promising applications are identified and an overview of recent developments in this domain is provided, including work on whole-body or hand motion and the various methodologies.
IEEE Transactions on Pattern Analysis and Machine IntelligenceAutomatic analysis of facial expressions: the state of the art
1,764 Citations2000M. Pantie, Léon Rothkrantz
The capability of the human visual system with respect to these problems is discussed, and it is meant to serve as an ultimate goal and a guide for determining recommendations for development of an automatic facial expression analyzer.
Computer Vision and Image UnderstandingA Survey of Computer Vision-Based Human Motion Capture
1,755 Citations2001Thomas B. Moeslund, Erik Granum
A comprehensive survey of computer vision-based human motion capture literature from the past two decades is presented, with a general overview based on a taxonomy of system functionalities, broken down into four processes: initialization, tracking, pose estimation, and recognition.
Pattern Recognition LettersInformation fusion in biometrics
1,566 Citations2003Arun Ross, Anil K. Jain
This paper addresses the problem of information fusion in biometric verification systems by combining information at the matching score level by combining three biometric modalities (face, fingerprint and hand geometry).
“Put-that-there”
1,432 Citations1980Richard A. Bolt
The work described herein involves the user commanding simple shapes about a large-screen graphics display surface, and because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression.
Human motion analysis: a review
1,408 Citations2002J.K. Aggarwal, Q. Cai
The paper gives an overview of the various tasks involved in motion analysis of the human body, and focuses on three major areas related to interpreting human motion: motion analysis involving human body parts, tracking of human motion using single or multiple cameras, and recognizing human activities from image sequences.
Choice Reviews OnlineThe Human-computer interaction handbook: fundamentals, evolving technologies, and emerging applications
1,359 Citations2013Juliele A. Jacko
This book discusses the evolution of Human-Computer Interaction in the age of Experience Ecosystems, as well as design principles for Usable Interfaces and Privacy, Security, and Trust.
Human-Computer Interaction
1,237 Citations2003
A visual task modeling language that can be used by system designers to model human computer interactions and can be translated into SpecTRM-RL, a blackbox specification language for modeling the automated portion of the control system.
Computer Vision and Image UnderstandingHuman Motion Analysis: A Review
1,145 Citations1999J.K. Aggarwal, Q. Cai
The Journal of the Acoustical Society of AmericaToward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion
1,093 Citations1993Iain R. Murray, John L. Arnott
The voice parameters affected by emotion are found to be of three main types: voice quality, utterance timing, and utterance pitch contour.
Behavior Research Methods, Instruments, & ComputersA breadth-first survey of eye-tracking applications
1,047 Citations2002Andrew T. Duchowski
Eye-tracking applications are surveyed in a breadth-first manner, reporting on work from the following domains: neuroscience, psychology, industrial engineering and human factors, marketing/advertising, and computer science.
Pattern RecognitionRecent developments in human motion analysis
962 Citations2003Liang Wang, Weiming Hu +1 more
This paper provides a comprehensive survey of research on computer-vision-based human motion analysis, namely human detection, tracking and activity understanding, and various methods for each issue are discussed in order to examine the state of the art.
Bricks
960 Citations1995George Fitzmaurice, Hiroshi Ishii +1 more
This work introduces the concept of Graspable User Interfaces that allow direct control of electronic or virtual objects through physical handles for control, and presents a design space for Bricks which lay the foundation for further exploring and developing Graspables User Inter interfaces.
International Journal of Human-Computer StudiesAffective computing: challenges
937 Citations2003Rosalind W. Picard
This article raises and responds to several criticisms of affective computing, articulating state-of-the art research challenges, especially with respect to affect in human-computer interaction.
Computer Vision and Image UnderstandingFace Detection: A Survey
917 Citations2001Erik Hjelmås, Boon Low
A comprehensive and critical survey of face detection algorithms, ranging from simple edge-based algorithms to composite high-level approaches utilizing advanced pattern recognition methods, is presented.
IEEE Transactions on Pattern Analysis and Machine IntelligenceCoding, analysis, interpretation, and recognition of facial expressions
914 Citations1997Irfan Essa, Alex Pentland
Computer Vision and Image UnderstandingFacial expression recognition from video sequences: temporal and static modeling
873 Citations2003Ira L. Cohen, Nicu Sebe +3 more
This work introduces and test different Bayesian network classifiers for classifying expressions from video, focusing on changes in distribution assumptions, and feature dependency structures, and proposes a new architecture of hidden Markov models (HMMs) for automatically segmenting and recognizing human facial expression from video sequences.
ACM Transactions on Information SystemsThe use of eye movements in human-computer interaction techniques
831 Citations1991Robert J. K. Jacob
Some of the human factors and technical considerations that arise in trying to use eye movements as an input medium are discussed and the first eye movement-based interaction techniques that are devised and implemented in the laboratory are described.
Wiley Encyclopedia of Computer Science and EngineeringHuman–Computer Interaction
804 Citations2009Jeremy R. Cooperstock
Human–computer interaction (HCI) plays an increasingly important role in ensuring that computer systems are not only functional but also respect the needs and capabilities of the humans that use them.
Proceedings of the IEEEToward an affect-sensitive multimodal human-computer interaction
784 Citations2003Maja Pantić, Léon Rothkrantz
It is argued that next-generation human-computer interaction (HCI) designs need to include the essence of emotional intelligence - the ability to recognize a user's affective states- in order to become more human-like, more effective, and more efficient.
Communications of the ACMTen myths of multimodal interaction
651 Citations1999Sharon Oviatt
Well-designed multimodal systems integrate complementary modalities to yield a highly synergistic blend in which the strengths of each mode are capitalized upon and used to overcome weaknesses in the other.
The logic of typed feature structures
607 Citations1992Bob Carpenter
IEEE Transactions on MultimediaAffective video content representation and modeling
593 Citations2005Alan Hanjalić, Li-Qun Xu
A computational framework for affective video content representation and modeling is proposed based on the dimensional approach to affect that is known from the field of psychophysiology that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect).
Real-Time ImagingReal-Time Eye, Gaze, and Face Pose Tracking for Monitoring Driver Vigilance
550 Citations2002Qiang Ji
The system was tested in a simulating environment with subjects of different ethnic backgrounds, different genders, ages, with/without glasses, and under different illumination conditions, and it was found very robust, reliable and accurate.
Evaluation of eye gaze interaction
534 Citations2000Linda E. Sibert, Robert J. K. Jacob
Two experiments are presented that compare an interaction technique developed for object selection based on a where a person is looking with the most commonly used selection method using a mouse and find that the eye gaze interaction technique is faster than selection with a mouse.
Lecture notes in computer scienceVision-Based Gesture Recognition: A Review
516 Citations1999Ying Wu, Thomas S. Huang
A survey on recent vision-based gesture recognition approaches is given and methods of static hand posture and temporal gesture recognition are reviewed, along with some thoughts about future research directions.
Applied Artificial IntelligenceThe open agent architecture: A framework for building distributed software systems
496 Citations1999David L. Martin, Adam Cheyer +1 more
The Open Agent Architecture is structured so as to minimize the effort involved in creating new agents and "wrapping" legacy applications; to encourage the reuse of existing agents; and to allow for dynamism and flexibility in the makeup of agent communities.
Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion
461 Citations2002Michael J. Black, Yaser Yacoob
This paper explores the use of local parametrized models of image motion for recovering and recognizing the non-rigid and articulated motion of human faces and shows how expressions can be recognized from the local parametric motions in the presence of significant head motion.
QuickSet
461 Citations1997Philip R. Cohen, Michael Johnston +6 more
QuickSet: Multimodal Interaction for Distributed Applications Philip R. Cohen, Michael Johnston, David McGee, Sharon Oviatt, Jay Pittman, Ira Smith, Liang Chen and Josh Glow Center for Human Computer Communication Oregon Graduate Institute of Science and Technology.
IEEE Transactions on Pattern Analysis and Machine IntelligenceRecognizing human facial expressions from long image sequences using optical flow
457 Citations1996Yaser Yacoob, L.S. Davis
Low-oil-content, fully lubricated leather is produced by treatment of leather with water emulsions of mixtures of alkanolamine soaps, oils, surfactants having HLB values between 2 and 6, and coupling solvents.
International Journal of Human-Computer StudiesThe production and recognition of emotions in speech: features and algorithms
438 Citations2003Oudeyer Pierre-Yves
A technique which allows to continuously control both the age of a synthetic voice and the quantity of emotions that are expressed and the first large-scale data mining experiment about the automatic recognition of basic emotions in informal everyday short utterances is presented.
Cambridge University Press eBooksThe Logic of Typed Feature Structures
409 Citations1992Robert L. Carpenter
The Logic of Typed Feature Structures is the first monograph that brings all the main theoretical ideas into one place where they can be related and compared in a unified setting and is an indispensable compendium for the researcher or graduate student working on constraint-based grammatical formalisms.
IEEE Transactions on Pattern Analysis and Machine IntelligenceLooking at people: sensing for ubiquitous and wearable computing
403 Citations2000Alex Pentland
The paper examines the mathematical tools that have proven successful, provides a taxonomy of the problem domain, and then examines the state of the art: person identification, surveillance/monitoring, 3D methods, and smart rooms/perceptual user interfaces.
Computers & Mathematics with ApplicationsLeonardo's laptop: Human needs and the new computing technologies
386 Citations2003
Human-Computer InteractionDesigning the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions
367 Citations2000Sharon Oviatt, Philip R. Cohen +9 more
The emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner are summarized-including early and late fusion approaches, and the new hybrid symbolic-statistical approach.
IEEE Transactions on Pattern Analysis and Machine IntelligenceAutomatic analysis of multimodal group actions in meetings
346 Citations2005L. McCowan, Daniel Gática-Pérez +4 more
Experiments demonstrate the importance of taking interactions into account in modeling the group actions and show that the visual modality contains useful information, even for predominantly audio-based events, motivating a multimodal approach to meeting analysis.
Intelligent multimedia interfaces
344 Citations1994Mark T. Maybury
The purpose of the workshop was to report on current advances in intelligent multimedia interface systems and their underlying theories, foster scientific interchange among these individuals, and evaluate current efforts and make recommendations for future investigations.
Human-Computer InteractionGestures Over Video Streams to Support Remote Collaboration on Physical Tasks
324 Citations2004Susan R. Fussell, Leslie D. Setlock +4 more
The results suggest that simple surrogate gesture tools can be used to convey gestures from remote sites, but that the tools need to be able to convey representational as well as pointing gestures to be effective.
IEEE Transactions on Intelligent Transportation SystemsDetermining driver visual attention with one camera
316 Citations2003Paul Smith, Mubarak Shah +1 more
This paper presents a system for analyzing human driver visual attention that relies on estimation of global motion and color statistics to robustly track a person's head and facial features and is able to track through yawning, which is a large local mouth motion.
"Eigenlips" for robust speech recognition
314 Citations2002Christoph Bregler, Yochai Konig
This study improves the performance of a hybrid connectionist speech recognition system by incorporating visual information about the corresponding lip movements by using a new visual front end, and an alternative architecture for combining the visual and acoustic information.
Mutual disambiguation of recognition errors in a multimodel architecture
314 Citations1999Sharon Oviatt
Although speech recognition as a stand-alone performed farmore poorly for accented speakers, their multimodal recognitionrates did not differ from those of native speakers, and implications are discussed for the development of future multi-modal architecture that can perform in a more robust and stable manner.
A design space for multimodal systems
304 Citations1993Laurence Nigay, Joëlle Coutaz
This paper presents an analysis of the integration of multiple communication modalities within an interactive system and presents a software architecture model of multimodal systems which supports these two salient properties: concurrency of processing and data fusion.
Communications of the ACMNoncommand user interfaces
304 Citations1993Jakob Nielsen
The next generation of UIs may move beyond the standard WIMP paradigm to involve elements such as virtual realities, head-mounted displays, sound and speech, pen and gesture recognition, animation and multimedia, limited artificial intelligence, and highly portable computers with cellular or other wireless communication capabilities.
Communications of the ACMTaming recognition errors with a multimodal interface
299 Citations2000Sharon Oviatt
My focus here is recognition errors as a problem for spoken-language systems, especially when processing diverse speaker styles or speech produced in noisy field settings, but when speech is combined with another input mode within a multimodal architecture, it is shown that two modes can function better than one alone.
IEEE Transactions on Neural NetworksHuman expression recognition from motion using a radial basis function network architecture
285 Citations1996Mark L. Rosenblum, Yaser Yacoob +1 more
A radial basis function network architecture is developed that learns the correlation of facial feature motion patterns and human expressions through a hierarchical approach which at the highest level identifies expressions, at the mid level determines motion of facial features, and at the low level recovers motion directions.
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
284 Citations2005Rana el Kaliouby, Peter Robinson
The system is based on a multi-level dynamic Bayesian network classifier which models complex mental states as a number of interacting facial and head displays, identified from component-based facial features.
Communications of the ACMPerceptual user interfaces: multimodal interfaces that process what comes naturally
274 Citations2000Sharon Oviatt, Philip R. Cohen
State-of-the-art multimodal speech and gesture systems now process complex gestural input other than pointing, and new systems have been extended to process different mode combinations—the most noteworthy being speech and pen input, and speech and lip movements.
Direct manipulation for comprehensible, predictable and controllable user interfaces
271 Citations1997Ben Shneiderman
Dramatic opportunities exist to develop direct manipulation programming to create end-user programming tools, dynamic queries to perform information search in large databases, and information visualization to support network database browsing.
Multimodal 'eyes-free' interaction techniques for wearable devices
259 Citations2003Stephen Brewster, Jo Lumsden +3 more
Two multimodal interaction techniques designed to overcome problems and allow truly mobile, 'eyes-free' device use are presented, one of which is a 3D audio radial pie menu that uses head gestures for selecting items.
Communications of the ACMGuidelines for multimodal user interface design
258 Citations2004Leah Reeves, Jennifer C. Lai +13 more
JMUI (Journal on Multimodal User Interfaces), Special issue “Best of affective computing and intelligent Guidelines for multimodal user interface design”, characteristics to the design of a user-oriented and guidelines of multimmodal interface design.
IEEE Transactions on Pattern Analysis and Machine IntelligenceSemisupervised learning of classifiers: theory, algorithms, and their application to human-computer interaction
252 Citations2004Ira L. Cohen, Fábio Gagliardi Cozman +3 more
A new analysis is provided that shows under what conditions unlabeled data can be used in learning to improve classification performance, and how the resulting algorithms are successfully employed in two applications related to human-computer interaction and pattern recognition: facial expression recognition and face detection.
ComputerSocially aware, computation and communication
252 Citations2005A. Pentlan
Technology must account for this by recognizing that communication is always socially situated and that discussions are not just words but part of a larger social dialogue, which forms a sort of collective intelligence.
Emerging Topics in Computer Vision
245 Citations2004Gérard Medioni, Sing Bing Kang
Gerard Medioni and Sing Bing Kang present advances in computer vision such as camera calibration, multi-view geometry, and face detection, and introduce important new topics such as vision for special effects and the tensor voting framework.
IEEE MultimediaAmbient intelligence: a multimedia perspective
227 Citations2004Emile Aarts
The concept of ambient intelligence is reviewed and its relation with multimedia is elaborated on and the emphasis is on qualitative aspects, highlighting those elements that play a role in realizing ambient intelligence.
IEEE Transactions on Pattern Analysis and Machine IntelligenceHuman activity recognition using multidimensional indexing
226 Citations2002J. Ben-Arie, Zhiqian Wang +2 more
A novel method for view-based recognition of human action/activity from videos that uses a sequence-based voting approach to recognize the activity invariant to the activity speed and finds that the probability of false alarm drops exponentially with the increased number of sampled body poses.
IEEE Signal Processing MagazineHand modeling, analysis and recognition
220 Citations2001Ying Wu, T.S. Huang
3-D hand models, various articulated motion analysis methods, and gesture recognition techniques employed in current research are studied, some of which are shown as examples.
Affective multimodal human-computer interaction
219 Citations2005Maja Pantić, Nicu Sebe +2 more
The components of human affect, how they might be integrated into computers, and how far are the authors from realizing affective multimodal human-computer interaction are discussed.
Proceedings of the National Academy of SciencesThe role of voice input for human-machine communication.
209 Citations1995Philip R. Cohen, Sharon Oviatt
This paper reviews application areas in which spoken interaction can play a significant role, assesses potential benefits of spoken interaction with machines, and compares voice with other modalities of human-computer interaction.
Emotion Recognition Based on Joint Visual and Audio Cues
203 Citations2006Nicu Sebe, Ira L. Cohen +2 more
The problem of bimodal emotion recognition is described and the use of probabilistic graphical models when fusing the different modalities is advocated.
Bimodal emotion recognition
198 Citations2002L.C. De Silva, Pei Chi Ng
The findings show that the audio and video information can be combined using a rule-based system to improve the recognition rate.
MULTIMODAL INTERFACES THAT PROCESS WHAT COMES NATURALLY
195 Citations2000Sharon Oviatt, Philip R. Cohen
WORLD SCIENTIFIC eBooksMULTIMODAL EMOTION RECOGNITION
191 Citations2005Nicu Sebe, Ira L. Cohen +1 more
New ways of human-computer interaction that enable the computer to be more aware of the user’s emotional and attentional expressions are explored.
A survey of research on context-aware homes
189 Citations2003Sven Meyer, Andry Rakotonirainy
Current research issues that need to be addressed are presented and strategies that facilitate the diffusion of information technology into homes in order to inspire positive emotions, encourage effortless exploration of content and help occupants to achieve tasks at hand are presented.
A living laboratory for the design and evaluation of ubiquitous computing technologies
185 Citations2005Stephen Intille, Kent Larson +4 more
The PlaceLab is a tool for researchers developing context-aware and ubiquitous interaction technologies that complements more traditional data gathering instruments and methods, such as home ethnography and laboratory studies.
Universal Access in the Information SocietyCommunication via eye blinks and eyebrow raises: video-based human-computer interfaces
184 Citations2003Kristen Grauman, Margrit Betke +3 more
Two video-based human-computer interaction tools are introduced that can activate a binary switch and issue a selection command that automatically detects a user’s eye blinks and accurately measures their durations.
A unified approach to coding and interpreting face images
180 Citations2002Andreas Lanitis, Chris Taylor +1 more
A compact parametrised model of facial appearance which takes into account all sources of variability and can be used for tasks such as image coding, person identification, pose recovery, gender recognition and expression recognition is described.
Proceedings of the IEEEIntegrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction
176 Citations2002Zoran Đurić, Wayne D. Gray +6 more
Technology and tools for intelligent human-computer interaction (IHCI) in which human cognitive, perceptual, motor and affective factors are modeled and used to adapt the H-C interface are described.
Unification-based multimodal integration
175 Citations1997Michael Johnston, Philip R. Cohen +4 more
A multimodal language processing architecture which supports interfaces allowing simultaneous input from speech and gesture recognition is described, which allows the component modalities to mutually compensate for each others' errors.
Courses and lecturesUser Modeling in Adaptive Interface
170 Citations1999Pat Langley
The notion of adaptive user interfaces, interactive systems that invoke machine learning to improve their interaction with humans, is examined and three ongoing research efforts that extend this framework in new directions are described.
Techniques for interactive audience participation
168 Citations2003Dan Maynes-Aminzade, Randy Pausch +1 more
A new set of techniques that enable members of an audience to participate, either cooperatively or competitively, in shared entertainment experiences, and can be implemented with inexpensive, off the shelf hardware.
IEEE Robotics & Automation MagazineA robotic wheelchair based on the integration of human and environmental observations - Look where you're going
168 Citations2003Yoshinori Kuno, Nobutaka Shimada +1 more
A robotic wheelchair that can understand the user's intentions from his/her behaviors and the environmental information and also observes the user when he/she is off the wheelchair, recognizing the user’s commands indicated by hand gestures is proposed.
Eye gaze estimation from a single image of one eye
166 Citations2003Wang, Sung +1 more
The two key contributions are that the possibility of finding the unique eye gaze direction from a single image of one eye is shown and that one can obtain better accuracy as a consequence of this.
The Journal of Rehabilitation Research and DevelopmentThe Smart Wheelchair Component System
164 Citations2004Richard C. Simpson, Edmund LoPresti +3 more
Tactile interfaces: a state-of-the-art survey
163 Citations2004M. Benali-Khoudja, Moustapha Hafez +2 more
This paper reports the progress in tactile interfaces technology in the following areas: teleoperation, telepresence, sensory substitution, 3D surface generation, Braille systems, laboratory prototypes, and games.
Communications of the ACMPerceptual user interfaces (introduction)
163 Citations2000Matthew Turk, George Robertson
NATO ASI series. Series F : Computer and system sciencesOn the Integration of Auditory and Visual Parameters in an HMM-based ASR
160 Citations1996A. Adjoudani, Christian Benoı̂t
A model which can improve the performances of an audio-visual speech recognizer in an isolated word and speaker dependent situation is proposed by using a hybrid system based on two HMMs trained respectively with acoustic and optic data.
Robust clustering of eye movement recordings for quantification of visual interest
156 Citations2004Anthony Santella, Doug DeCarlo
An automatic data-driven method is presented, which clusters visual point-of-regard (POR) measurements into gazes and regions- of-interest using the mean shift procedure, which forms a structured representation of viewer interest.
Real-Time ImagingVideo analysis of human dynamics—a survey
150 Citations2003Jessica JunLin Wang, Sameer Singh
This paper provides a detailed survey of the various studies in areas related to the tracking of people and body parts such as face, hands, fingers, legs, etc., and modeling behavior using motion analysis.
Conversing with the user based on eye-gaze patterns
147 Citations2005Pernilla Qvarfordt, Shumin Zhai
It is demonstrated that eye-gaze could play an important role in managing future multimodal human-computer dialogues.
…