Researchers develop tools to turn audio clips into realistic video

Researchers have developed new computer algorithms that turn audio clips into a realistic, lip-synced video of the person speaking those words.

The team successfully generated a highly realistic video of former US President Barack Obama talking about terrorism, fatherhood, job creation and other topics using audio clips of those speeches and existing weekly video addresses that were originally on a different topic.

"These type of results have never been shown before," said Ira Kemelmacher- Shlizerman, Assistant Professor at the University of Washington's Paul G. Allen School of Computer Science & Engineering.

"Realistic audio-to-video conversion has practical applications like improving video conferencing for meetings, as well as futuristic ones such as being able to hold a conversation with a historical figure in virtual reality by creating visuals just from audio. This is the kind of breakthrough that will help enable those next steps," Kemelmacher-Shlizerman said of the research to be presented at SIGGRAPH 2017 in Los Angeles.

Previously, audio-to-video conversion processes have involved filming multiple people in a studio saying the same sentences over and over to try to capture how a particular sound correlates to different mouth shapes, which is expensive, tedious and time-consuming.

By contrast, lead study author Supasorn Suwajanakorn, a recent doctoral graduate in the Allen School, developed algorithms that can learn from videos that exist "in the wild" on the Internet or elsewhere.

"There are millions of hours of video that already exist from interviews, video chats, movies, television programmes and other sources. And these deep learning algorithms are very data hungry, so it's a good match to do it this way," Suwajanakorn said.

In a visual form of lip-syncing, the system converts audio files of an individual's speech into realistic mouth shapes, which are then grafted onto and blended with the head of that person from another existing video.

The team chose Obama because the machine learning technique needs available video of the person to learn from, and there were hours of presidential videos in the public domain.

"In the future video, chat tools like Skype or Messenger will enable anyone to collect videos that could be used to train computer models," Kemelmacher-Shlizerman said.

Because streaming audio over the Internet takes up far less bandwidth than video, the new system has the potential to end video chats that are constantly timing out from poor connections.

"When you watch Skype or Google Hangouts, often the connection is stuttery and low-resolution and really unpleasant, but often the audio is pretty good," said co-author Steve Seitz, Professor at the University of Washington.

"So if you could use the audio to produce much higher-quality video, that would be terrific," Seitz said.

--IANS

gb/bg