New technique recovers spoken words from video of objects in room

A new research has detailed a simple optical technique by which audio information can be extracted from high-speed video recordings.

Reiner Eschbach from Xerox Corp said that one of the intriguing aspects of the study is the ability to recover spoken words from a video of objects in the room.

Eschbach added that the research shows that the sound creates minute vibrations in objects and that these vibrations given the right equipment can be picked up from a video signal, which is an interesting foray into a new application space and will trigger interesting research in the field.

The technique is based on the fact that sound waves are mechanical waves that cause air to vibrate when traveling, the paper notes and that vibration through air can cause vibration of objects located in its traveling path, especially if the objects are lightweight, thin, and flexible, such as a piece of paper.

The vibrations, although usually with small amplitudes, can be detected and analyzed algorithmically, and audio reconstructed based on those calculations.

The authors used a subset-based image-correlation approach to detect the motions of points on the surface of an object, capturing target images with a high-speed camera and applying the Gauss-Newton algorithm and a few other measures to achieve very fast and highly accurate image matching.

Because the detected vibrations are directly related to sound waves, a simple model was used to reconstruct the original audio information of the sound waves.

The study is published in the journal Optical Engineering.