Wav2Mov

Speech ControlledFacial Animation using Generative Adversarial Networks

wav2mov | Home

Introduction

Speech-driven facial animation is the process which uses speech signals to automatically synthesize a talking character given a reference face image. The majority of work in this domain creates a mapping from audio features to visual features.

Creating facial animation often requires post-processing using computer graphics techniques to produce realistic subject dependent results.Lip Syncing from the information available from speech input alone can boost the developments in wide range of fields such as animation ,film dubbing and advertising industry.

In this work we propose a neural network based speech to facial animation based system having Generative Adversarial Networks at its core. We propose an end to end system for achieving lip synchronization with given speech input and a reference image to generate the video frames.

Applications

  • Animation and game industries : Animations take a huge part in entertainment aspects. As humans are more sensitive at picking up abnormalities in facial motion and lip syncing , it becomes crucial to achieve realistic characters. The process can be automated using neural network based systems which can achieve such lip syncing of animated characters with only speech input
  • Movie dubbing : People tend to watch movies of genres of their interest and which may be not always in the same native language. So dubbing plays an important role.But it requires lot of time to do that ,although lip sync is not achieved. But using speech to facial animation systems lip syncing can be easily incorporated to such films.
  • Video transfer over internet Band-limited visual telecommunications can be improved by either generating entire visual content based on audio or filling in dropped frames. Moreover, they can be used to generate parts of the face that are occluded or missing in a scene.
  • Other applications include animating faces during voice/video calls , and other entertainment based mobile/web applications.