Meta details AI system capable of generating 3D visuals from text prompts

The Meta 3D Gen system takes user input in text form and creates a 3D mesh structure and adds texture and other detailing

Meta, the parent company of Facebook, Instagram and Whatsapp, has published a research paper detailing “Meta 3D Gen AI” system, which it said is capable of generating 3D models using text prompt. Meta said that the 3D Gen is a combined AI system that can generate high-quality 3D assets, with both high-resolution textures and material maps, all within one minute.

Meta 3D Gen: What is it

According to the paper published by the Researchers at Meta, Meta 3D Gen is a combined system that uses two AI-powered sub-systems called Meta 3D AssetGen and Meta 3D TextureGen. The system can take text prompts from the user and produce a final output of AI generated 3D content such as characters, props or scenes. Additionally, the user can also provide the system with an already created 3D mesh, which is essentially the structure of a 3D model, and ask it to add texture to it.

Meta 3D Gen: How it works

Meta said that the 3D Gen AI system uses Physically-Based Rendering (PBR) techniques to build up 3D content from scratch. The system first takes an input from the user which contains instructions for creating a 3D structure and its design elements including the texture.

In the first step, the Meta 3D AssetGen system creates an initial “3D asset” based on the user prompt. This step produces a 3D mesh with texture and PBR material maps. Meta said that the inference time for this step is approximately 30 seconds.

In stage two, a higher-quality texture and PBR maps are generated for the 3D asset generated in stage one. This step utilises the Meta 3D TextureGen system that takes approximately 20 seconds, according to the company.

Additionally, the Meta 3D TextureGen system can be used individually by the user as a text-to-texture generator. If the user provides the system with an already generated 3D mesh along with a text prompt detailing the texture they want on the 3D content, the system automatically initiates step two.

In an example shared by the company, a user inputs a prompt asking the AI system to generate a T-rex wearing a green wool sweater, the system first creates a 3D visual for the T-rex and then adds the wool sweater texture and other colour details.

Meta 3D Gen (Source: Meta)