New software can edit human voice like text

Scientists have developed a new software that allows people to edit audio recording of a human voice with the ease of changing words on a computer screen.
The technology developed by researchers at Princeton University in the US may do for audio recordings what word processing software did for the written word.
The software, named VoCo, provides an easy means to add or replace a word in an audio recording of a human voice by editing a transcript of the recording.
New words are automatically synthesised in the speaker's voice even if they do not appear anywhere else in the recording.

The system, which uses a sophisticated algorithm to learn and recreate the sound of a particular voice, could one day make editing podcasts and narration in videos much easier.
The technology could provide a launching point for creating personalised robotic voices that sound natural.

"VoCo provides a peek at a very practical technology for editing audio tracks, but it is also a harbinger for future technologies that will allow the human voice to be synthesised and automated in remarkable ways," said Adam Finkelstein, a professor of computer science at Princeton.
On a computer screen, VoCo's user interface looks similar to other audio editing software such as the popular podcast editing programme Audacity or Apple's music editing programme GarageBand.

It offers visualisation of the waveform of the audio track and a set of cut, copy and paste tools for editing.
Unlike other programmes, however, VoCo also augments the waveform with a text transcript of the track and allows the user to replace or insert new words that do not already exist in the track simply by typing in the transcript.
When the user types the new word, VoCo updates the audio track, automatically synthesising the new word by stitching together snippets of audio from elsewhere in the narration.
"Currently, audio editors can cut out pieces of a track of narration and move a clip from one place to another," said Finkelstein.

"However, if you want to add a word that does not exist in the recording, it is possible only through a painstaking trial and error process of searching for small audio snippets that might fit together well enough to plausibly form the word," said Finkelstein.
"VoCo automates the search and stitching process, and produces results that typically sound even better than those created manually by audio experts," he said.
At the heart of VoCo is an optimisation algorithm that searches the voice recording and chooses the best possible combinations of partial word sounds, called "phonemes," to build new words in the user's voice.

To do this, it not only needs to find the individual phonemes, but also find sequences of them that stitch together without abrupt transitions, as well as fit them into the existing sentence so that the new word blends in seamlessly.
Words are pronounced with different emphasis and intonation depending on where they fall in a sentence, so context is important.
For clues about this context, VoCo looks to an audio track of the sentence that is automatically synthesised in artificial voice from the text transcript - one that sounds robotic to human ears.
This recording is used as a point of reference in building the new word. VoCo then matches the pieces of sound from the real human voice recording to match the word in the synthesised track - a technique known as "voice conversion," which inspired the project name VoCo.

The finding was published in the journal Transactions on Graphics.