What are AI neolabs and why are they betting on safe superintelligence?

Superstar AI researchers are leaving tech giants to build independent research laboratories. Interestingly, one theme is emerging as central to the objectives of some of these labs

A growing number of artificial intelligence (AI) researchers are leaving established technology companies to set up independent research laboratories. Called AI neolabs, these ventures pursue long-term technical breakthroughs, with some prioritising research over immediate commercial returns.

 

The trend has attracted billions of dollars in investment worldwide. While these laboratories have different objectives, a common research area emerging across them is the drive to develop increasingly capable AI systems while ensuring they remain safe and aligned with human intentions.

The rise of neolabs: Billions in funding, few commercial products

According to a report by The Economist, 29 neolabs were founded over two years and raised a combined $23 billion. Less than 35 per cent had released commercial offerings, while only slightly more than half had published code, often in the form of research demonstrations.

 

 

Safe Superintelligence Inc (SSI), founded by former OpenAI chief scientist Ilya Sutskever in June 2024, is among the most prominent examples. The company raised $1 billion within four months of its launch and later reached a reported valuation of $32 billion. Nvidia subsequently invested another $5 billion, taking SSI's total reported funding to $8 billion, according to The Economist.

 

Interestingly, SSI's stated objective is to develop safe superintelligence. Its website said it has “one goal and one product: a safe superintelligence” and that it intends to advance AI capabilities while keeping safety ahead of them.

 

Meanwhile, Thinking Machines Lab, founded by former OpenAI chief technology officer Mira Murati, raised $2 billion at a $12 billion valuation, according to The Economist. While its objective is different from SSI's, the firm aims to build AI that extends human will and judgement. It has also launched Tinker, a platform that allows researchers and developers to customise and fine-tune AI models.

Why is superintelligence safety a research priority?

Superintelligence refers to AI systems that surpass human intellectual capabilities across a broad range of tasks. The challenge is not just developing such systems, but ensuring that they continue to follow human intentions as their capabilities increase.

 

This challenge is known as the AI alignment problem. Alignment means making sure that an AI system's actions remain consistent with human goals, values and intentions.

 

The unresolved alignment problem made headlines last month when Jacob Coxon, a researcher at AI company Anthropic, resigned and warned about the risks of developing increasingly powerful AI systems.

 

“The consensus is that the next year or two is crunch time for humanity,” Coxon told WIRED. He argued that competition between AI developers could make it harder to prioritise safety and called for greater coordination between companies and governments.

 

Researchers are yet to establish a comprehensive way to guarantee that advanced AI systems will behave as intended in unfamiliar situations. Testing can reveal failures, but it cannot prove that every significant safety problem has been eliminated.

 

This has led some research organisations to explore different approaches to alignment and control.

 

Conjecture, an AI alignment research startup, has investigated an architecture called Cognitive Emulation, designed to bound AI capabilities and make systems reason in ways humans can understand and control. The Institute for Responsible Superintelligence (RESI), a nonprofit research institute, is also pursuing safety mechanisms whose guarantees can be analysed before deployment.