Adaptive bimodal sensor fusion for automatic speechreading

Published 24 December 2002

Uwe Meier, Wolfgang Hürst, Paul Duchnowski

DOI10.1109/icassp.1996.543250

Citations76

TL;DR

Different methods of combining the visual and acoustic data to improve the recognition performance of automated speech recognizers by using additional visual information are presented, achieving error reduction of up to 50%.

Abstract

We present work on improving the performance of automated speech recognizers by using additional visual information: (lip-/speechreading); achieving error reduction of up to 50%. This paper focuses on different methods of combining the visual and acoustic data to improve the recognition performance. We show this on an extension of an existing state-of-the-art speech recognition system, a modular MS-TDNN. We have developed adaptive combination methods at several levels of the recognition network. Additional information such as estimated signal-to-noise ratio (SNR) is used in some cases. The results of the different combination methods are shown for clean speech and data with artificial noise (white, music, motor). The new combination methods adapt automatically to varying noise conditions making hand-tuned parameters unnecessary.

Keywords

Computer ScienceEngineering

NatureHearing lips and seeing voices

6,130 Citations1976Harry McGurk, John MacDonald

The study reported here demonstrates a previously unrecognised influence of vision upon speech perception, on being shown a film of a young woman's talking head in which repeated utterances of the syllable [ba] had been dubbed on to lip movements for [ga].

IEEE Transactions on Acoustics Speech and Signal ProcessingPhoneme recognition using time-delay neural networks

2,625 Citations1989Alexander Waibel, Toshiyuki Hanazawa +3 more

The authors present a time-delay neural network (TDNN) approach to phoneme recognition which is characterized by two important properties: using a three-layer arrangement of simple computing units, a hierarchy can be constructed that allows for the formation of arbitrary nonlinear decision surfaces, which the TDNN learns automatically using error backpropagation.

PharmacologyAutomatic lipreading to enhance speech recognition (speech reading)

328 Citations1984E. Petajan

An automatic lipreading system which has been developed and the combination of the acoustic and visual recognition candidates is shown to yield a final recognition accuracy which greatly exceeds the acoustic recognition accuracy alone.

The Journal of the Acoustical Society of AmericaIntermodal timing relations and audio-visual speech recognition by normal-hearing adults

203 Citations1985Matthew McGrath, Quentin Summerfield

It was concluded that most observers, whether good lipreaders or not, possess insufficient sensitivity to intermodal timing cues in audio-visual speech for them to be used analogously to voice onset time in auditory speech perception.

Systems and Computers in JapanAutomatic lipreading by optical‐flow analysis

165 Citations1991Kenji Mase, Alex Pentland

A computer system is developed for automatic recognition of continuously spoken words by using only visual data, and the pattern of muscle action is used to recognize the spoken words.

Visual Speech Recognition with Stochastic Networks

148 Citations1994Javier R. Movellan

The results indicate that simple hidden Markov models may be used to successfully recognize relatively unprocessed image sequences, and the system achieved performance levels equivalent to untrained humans when asked to recognize the first four English digits.

IEEE International Conference on Acoustics Speech and Signal ProcessingImproving connected letter recognition by lipreading

124 Citations1993Christoph Bregler, Hermann Hild +2 more

The authors show how recognition performance in automated speech perception can be significantly improved by additional lipreading, so called speech-reading, on an extension of a state-of-the-art speech recognition system, a modular multistage time delay neural network architecture (MS-TDNN).

Neural network lipreading system for improved speech recognition

108 Citations2003David G. Stork, Gregory J. Wolff +1 more

A modified time-delay neural network (TDNN) has been designed to perform both automatic lipreading (speech reading) in conjunction with acoustic speech recognition in order to improve recognition both in silent environments as well as in the presence of acoustic noise.

Toward movement-invariant automatic lip-reading and speech recognition

79 Citations2002Paul Duchnowski, M. Hunke +3 more

Journal of Speech Language and Hearing ResearchEffects of Consonantal Context on Vowel Lipreading

20 Citations1987Allen A. Montgomery, Brian E. Walden +1 more