Microsoft unveils generative AI tool that can create video-game scenes

By Dina Bass 

Microsoft Corp. unveiled artificial intelligence tools that can create video-game scenes that would normally have to be programmed and animated by a human, a model it built using data collected from Xbox gamers and their controllers.   

Called Muse, the model  is the first of its kind, according to Microsoft. A machine learning research team surveyed game developers to discern how generative AI could be helpful and what they’d need for such tools to be effective, said Katja Hofmann, a senior principal research manager at Microsoft. Then, to train the AI model, Hofmann’s team collected seven years’ worth of gameplay data from Bleeding Edge, a 2020 multiplayer battle game from Xbox’s Ninja Theory studio. 

 

“If you haven’t seen any of this generated gameplay footage before, you might think these are just clips from a video game,” Hofmann said in a demonstration for reporters. The Muse model was announced Wednesday in the journal Nature.

 

Microsoft’s Xbox team and other industry players are eager to use AI to help reduce the hundreds of millions of dollars studios typically spend on blockbusters — costs that have skyrocketed in recent years even as sales growth has slowed. It’s controversial because players are suspicious that generative AI could hurt game quality. And studio employees, already hit with a series of layoffs, fret that the technology could further reduce their ranks. 

 

Bleeding Edge pits two teams of four against each another and is played entirely online. All games are recorded with players’ permission. That let the researchers amass more than a billion game images paired with corresponding controller usage.  The result is what Microsoft calls a “World and Human Action Model.” Like most generative AI models, it creates something after being prompted — in this case a sequence of visuals after being shown a game scene created by a human.

 

“The model is able to generate something that is really consistent, that demonstrates a very accurate understanding of the actual game world,” Hofmann said. The AI-made elements include interactions between characters and other elements of the game. 

 

Outside researchers will be able to build on Microsoft’s work with prototype software the company plans to release.