Realtime face analysis and synthesis using neural network

Abstract

Describes recent research results about how to generate an avatar's face in a real-time process, exactly copying a real person's face. It is very important for the synthesis of a real avatar to precisely duplicate the emotions and impressions included in the original face image and voice. A face-fitting tool from multi-angle camera images is introduced in order to make a real 3D face model with real texture and geometry that is very close to the original. When an avatar is speaking something, the voice signal is very essential for deciding the mouth shape features, so a real-time mouth shape control mechanism is proposed by conversion from speech parameters to lip shape parameters using a multi-layered neural network. For dynamic modeling of facial expressions, a muscle structure constraint is introduced to generate a facial expression naturally with just a few parameters. We also tried to obtain muscle parameters automatically in order to decide an expression from a local motion vector on the face calculated by optical flow in a video sequence. Finally, an approach is presented that enables the modeling of the emotions appearing on faces. A system with this approach helps us to analyze, synthesize and code face images at the emotional level.