Disney's 'Magic Bench' lets users see, feel animations

Disney researchers have created a 'Magic Bench' that lets you drift through a magical world where you can see, hear and feel animated characters, by combining augmented and mixed reality experience.
Sitting on the Magic Bench you may have an elephant hand you a glowing orb or you might get rained on or a tiny donkey might saunter by and kick the bench, researchers said.
It does not involve wearing a head-mounted display or using a handheld device. In this experience the surroundings are instrumented rather than the individual, allowing people to share the magical experience as a group.
People seated on the Magic Bench can see themselves in a mirrored image on a large screen in front of them, creating a third person point of view.

The scene is reconstructed using a depth sensor, allowing the participants to actually occupy the same 3D space as a computer-generated character or object, rather than superimposing one video feed onto another.
"This platform creates a multi-sensory immersive experience in which a group can interact directly with an animated character. Our mantra for this project was: hear a character coming, see them enter the space, and feel them sit next to you," said Moshe Mahler, principal digital artist at Disney Research.

Researchers used a colour camera and depth sensor to create a real-time, HD-video-textured 3D reconstruction of the bench, surroundings, and participants.

The algorithm reconstructs the scene, aligning the RGB (red green blue) camera information with the depth sensor information.
To eliminate depth shadows that occur in areas where the depth sensor has no corresponding line of sight with the colour camera, a modified algorithm creates a 2D backdrop.
The 3D and 2D reconstructions are positioned in virtual space and populated with 3D characters and effects in such a way that the resulting real-time rendering is a seamless composite, fully capable of interacting with virtual physics, light, and shadows.
"The bench itself plays a critical role. Not only does it contain haptic actuators, but it constrains several issues for us in an elegant way," Mahler said.

Researchers noted that the location and the number of participants, and can infer their gaze. It creates a stage with a foreground and a background, with the seated participants in the middle ground.
"It even serves as a controller; the mixed reality experience doesn't begin until someone sits down and different formations of people seated create different types of experiences," Mahler added.