Novel robot learns to play Jenga

MIT scientists have developed a novel robot that can play a game of Jenga, combining vision and touch to carefully extract blocks without toppling the tower.

In the game of Jenga, 54 rectangular blocks are stacked in 18 layers of three blocks each, with the blocks in each layer oriented perpendicular to the blocks below.

The aim of the game is to carefully extract a block and place it at the top of the tower, thus building a new level, without toppling the entire structure.

The robot, described in the journal Science Robotics, is equipped with a soft-pronged gripper, a force-sensing wrist cuff, and an external camera, all of which it uses to see and feel the tower and its individual blocks.

As the robot carefully pushes against a block, a computer takes in visual and tactile feedback from its camera and cuff, and compares these measurements to moves that the robot previously made.

It also considers the outcomes of those moves -- specifically, whether a block, in a certain configuration and pushed with a certain amount of force, was successfully extracted or not.

In real-time, the robot then "learns" whether to keep pushing or move to a new block, in order to keep the tower from falling.

"Unlike in more purely cognitive tasks or games such as chess or Go, playing the game of Jenga also requires mastery of physical skills such as probing, pushing, pulling, placing, and aligning pieces," said Alberto Rodriguez, an assistant professor at Massachusetts Institute of Technology (MIT) in the US.

"It requires interactive perception and manipulation, where you have to go and touch the tower to learn how and when to move blocks," Rodriguez said.

"This is very difficult to simulate, so the robot has to learn in the real world, by interacting with the real Jenga tower. The key challenge is to learn from a relatively small number of experiments by exploiting common sense about objects and physics," he said.

The tactile learning system the researchers have developed can be used in applications beyond Jenga, especially in tasks that need careful physical interaction, including separating recyclable objects from landfill trash and assembling consumer products.

"In a cellphone assembly line, in almost every single step, the feeling of a snap-fit, or a threaded screw, is coming from force and touch rather than vision," Rodriguez said.

"Learning models for those actions is prime real-estate for this kind of technology," he said.

Curious as to how their machine-learning approach stacks up against actual human players, the team carried out a few informal trials with several volunteers.

"We saw how many blocks a human was able to extract before the tower fell, and the difference was not that much," said research team member Miquel Oller.

However, there is still a way to go if the researchers want to competitively pit their robot against a human player. In addition to physical interactions, Jenga requires strategy, such as extracting just the right block that will make it difficult for an opponent to pull out the next block without toppling the tower.

For now, the team is less interested in developing a robotic Jenga champion, and more focused on applying the robot's new skills to other application domains.