New Disney app turns coloured drawings into 3D characters

A colouring book app devised by researchers at Disney can cause characters from drawings to leap from the page in 3-D with the help of augmented reality.
A child colours a character, such as an elephant, on the book page normally, while a tablet or smartphone running the new app monitors the drawing.
Based on the child's colouring, the app fills in colours in real-time on an animated 3-D version of the elephant that is visible on the device's screen and integrated into the video.
The app keeps the core focus on the traditional activity of colouring while offering a magical digital overlay that enhances engagement.

In user testing - performed with adults rather than children in the study - the researchers found that most users said the app increased their motivation to draw in colouring books and 80 per cent said the app increased their feeling of connection to a character.
To create the new experience, the researchers first created animated 3-D virtual characters and then used custom software to generate 2-D line-art representations of the characters for a colouring book.

The app, operating on a device with a camera viewing the user and the colouring book, automatically detects the character the user is colouring and displays the 3-D version.

As the child applies colour to the 2-D drawing, the app applies the same colour to the 3-D character - both to the areas visible in the 2-D drawing and to the remainder of the 3-D form not visible in the book.
Because the colouring occurs in real-time, the illusion is created that the user is also colouring the occluded areas, with similar texturing of the colour.
Determining how to apply colour to the occluded areas was one of the tougher problems to solve, said Robert W Sumner, a principal research scientist who leads the group on animation and interactive graphics at Disney Research.

Simply mirroring the user's coloured strokes doesn't work because the pattern of colours used for, say, a character's face will not be appropriate for the back of the character's head.
The colour also has to be continuous, so no seams can be seen between the visible areas and the occluded areas or where disparate portions of the textures meet.
Their approach was to create a "lookup map" for each character, which matches pixels in the occluded parts with corresponding pixels in the portion visible to the user.
User testing showed that this method provides better results than "naive" approaches such as mirroring. Most importantly, the lookup map method enables the colouring to be performed instantly.