Speech-driven 3D face animation technique, extending its applications to various multimedia fields. Previous research has generated promising realistic lip movements and facial expressions from audio signals. However, traditional regression models solely driven by data face several essential problems, such as difficulties in accessing precise labels and domain gaps between different modalities, leading to unsatisfactory results lacking precision and coherence.

To enhance the visual accuracy of generated lip movement while reducing the dependence on labeled data, we propose a novel framework SelfTalk, by involving self-supervision in a cross-modals network system to learn 3D talking faces. The framework constructs a network system consisting of three modules: facial animator, speech recognizer, and lip-reading interpreter. The core of SelfTalk is a commutative training diagram that facilitates compatible features exchange among audio, text, and lip shape, enabling our models to learn the intricate connection between these factors. The proposed framework leverages the knowledge learned from the lip-reading interpreter to generate more plausible lip shapes. Extensive experiments and user studies demonstrate that our proposed approach achieves state-of-the-art performance both qualitatively and quantitatively. We recommend watching the supplementary.

Overview of the proposed SelfTalk. We constructed a commutative training diagram consisting of three modules: facial animator, speech recognizer, and lip-reading interpreter. Specifically, given an input audio signal , the facial animator module extracts the corresponding facial animation , which constitutes the core component of our framework. The speech recognizer, on the other hand, is capable of producing the corresponding text and utilizing it as a ground truth label for supervision. Lastly, the lip-reading interpreter interprets lip movements, produces a text distribution and establishes a constraint using the label from .