AI system could make lip sync dubbing accurate

Scientists have developed an artificial intelligence (AI) system that can edit the facial expressions of actors to accurately match dubbed voices, saving time and reducing costs for the film industry.

Deep Video Portraits, developed by researchers including those from Max Planck Institute for Informatics in Germany and Stanford University in the US can also be used to correct gaze and head pose in video conferencing, and enables new possibilities for video postproduction and visual effects.

Unlike previous methods that are focused on movements of the face interior only, Deep Video Portraits can also animate the whole face including eyes, eyebrows, and head position in videos, using controls known from computer graphics face animation.

It can even synthesise a plausible static video background if the head is moved around.

"It works by using model-based 3D face performance capture to record the detailed movements of the eyebrows, mouth, nose, and head position of the dubbing actor in a video," said Hyeongwoo Kim from University of Bath in the UK.

"It then transposes these movements onto the 'target' actor in the film to accurately sync the lips and facial movements with the new audio," said Kim.

The research is currently at the proof-of-concept stage and is yet to work at real time, however the researchers anticipate the approach could make a real difference to the visual entertainment industry.

"Despite extensive post-production manipulation, dubbing films into foreign languages always presents a mismatch between the actor on screen and the dubbed voice," said Christian Theobalt, from the Max Planck Institute for Informatics.

"Our new Deep Video Portrait approach enables us to modify the appearance of a target actor by transferring head pose, facial expressions, and eye motion with a high level of realism," said Theobalt.

"This technique could also be used for post-production in the film industry where computer graphics editing of faces is already widely used in today's feature films," said Christian Richardt, from the University of Bath.0