What is superalignment and why are AI researchers raising concerns?

Anthropic researcher Jacob Coxon's warning about self-improving AI has raised concerns over superalignment. Here's what it means and why it matters

Today, using artificial intelligence (AI) can feel as simple as giving an instruction. We open ChatGPT or Claude, type a prompt, get the task done, and if the answer is not good enough, ask the system to try again. We are still the ones giving the instructions. But what happens when that equation begins to reverse? What happens when AI systems become more capable than the people supervising them, and keeping control over them becomes a challenge?

 

This challenge is known as superalignment. It was in focus this week after an AI researcher at Anthropic resigned from the company, warning that the industry was “racing straight to self-improving superintelligence”.

 

 

In a series of posts on X, Coxon, who has previously worked at OpenAI, accused both companies of “irresponsible” behaviour and said the technology could eventually “kill us all”. He warned of the emergence of “superhuman systems” that could “hack anything, revolutionise any field overnight, and acquire real power and resources”.

 

Reposting Coxon’s comments, Anthropic researcher Evan Hubinger said: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

 

“What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought,” he added.

 

Soon, several researchers from leading AI firms and labs started to weigh in, and most of them agreed with Coxon’s concerns.

 

But what exactly is the alignment problem in AI, and why does Hubinger's warning about recursive self-improvement matter?

What does alignment mean in artificial intelligence?

Alignment, in terms of artificial intelligence, means ensuring that an AI system’s behaviour matches human intentions, values and goals. It is about making sure that the system follows what humans intend, rather than simply following an instruction literally.

 

Superalignment is an extension of the traditional AI alignment problem. As AI companies work towards increasingly capable systems and, eventually, superintelligence — a stage at which AI could exceed human intelligence — researchers face the challenge of ensuring that such systems can still be reliably supervised, steered and controlled.

 

If an AI system becomes substantially more capable than its human supervisors, humans may no longer be able to fully evaluate its reasoning, determine whether its actions are safe, or identify when its behaviour diverges from their intentions.

Why is recursive self-improvement raising concerns?

When Coxon said AI is “racing straight to self-improving superintelligence”, the concern is not simply that AI systems will become more intelligent, but that they could eventually acquire the ability to improve or develop newer versions of themselves without human intervention.

 

This is known as recursive self-improvement. Such a capability does not exist today, but researchers have warned that if an AI system could repeatedly improve its own capabilities, the process could accelerate rapidly and make it increasingly difficult for humans to retain control over the system.

Why has the superalignment problem not been solved?

The difficulty is that researchers do not yet have a proven method to ensure that increasingly intelligent systems will remain aligned as their capabilities exceed those of their human supervisors.

 

Jakub Pachocki, chief scientist at OpenAI, acknowledged the problem in a blog published a few days ago. “Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer,” he said.

 

“I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world,” he added.

 

One possible approach suggested by Pachocki is to build automated AI researchers that can work on the alignment problem and improve iteratively, while keeping humans involved in the AI self-improvement loop.

 

At the same time, he said the benefits of increasingly intelligent machines could be spread through faster scientific progress and economic growth, as well as by giving individuals access to their own artificial general intelligence. This would allow them to benefit directly from the capabilities of highly intelligent systems.

 

The immediate concern is not that superintelligence already exists, but that progress towards increasingly capable AI systems may be moving faster than the development of methods to control them.

 

According to Pachocki, the focus during this transition to highly intelligent machines should be on preserving human agency, preventing excessive concentration of power and ensuring that humans remain in control of their future.

 

However, Coxon and other researchers have argued that the industry is not doing enough to address those challenges with sufficient urgency.