OpenAI board member says firm not doing enough, warns of catastrophic risks

Paul Christiano, who is a technology adviser to the US government, said the AI industry is not doing enough to reduce the risks posed by AI to an acceptable level.

A day after joining OpenAI’s nonprofit board, Paul Christiano said the rapid acceleration of artificial intelligence could lead to a “catastrophic and irreversible loss of control in the very near term”.

 

Christiano, who is a technology adviser to the US government, said the AI industry, including OpenAI, is not doing enough to reduce the risk to an acceptable level.

 

“Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level,” Christiano wrote in a statement shared on X.

 

 

The technology adviser will serve on OpenAI’s Safety and Security Committee to support safety oversight. Christiano will also serve as a non-voting observer on the board of OpenAI Group PBC, the firm’s commercial arm.

 

Christiano was previously associated with OpenAI from 2017 to 2021, when he served as a member of the technical staff. He studied for a bachelor’s degree in mathematics at the Massachusetts Institute of Technology.

 

“I’m joining because I believe that if OpenAI rises to the occasion, we could significantly reduce risk,” Christiano said.

 

He added, “My joining is not an endorsement or criticism of OpenAI’s safety practices in particular; I hope that all frontier companies strengthen safety oversight, and I am excited to work on this at OpenAI. I believe that the rest of the world should judge OpenAI, and all AI developers, by externally verifiable behavior and results.”

 

Christiano’s appointment comes amid renewed scrutiny of OpenAI’s safety procedures, following a series of incidents in which its AI agents went out of control and penetrated external computer systems without the knowledge of the company’s researchers.

 

On Tuesday, Anthropic researcher Jacob Coxon also resigned from his position, saying that the company and its competitors are racing to build AI systems that they “may not be able to control”. 

What did Christiano say?

 

Christiano said the risk of losing control of AI has become “acute”, citing the possibility of rapid advances driven by automated AI research and development.

 

AI systems could soon fully automate AI research, creating a feedback loop in which improvements to training and algorithms produce better AI researchers capable of making further advances, Christiano said. While the evidence remains uncertain, this could overcome diminishing returns and computing bottlenecks, triggering a rapid “intelligence explosion”, he added.

 

If AI R&D becomes fully automated, Christiano said, the resulting acceleration could produce more algorithmic progress within six months than has occurred since the development of the Transformer nearly a decade ago, potentially resulting in superintelligent AI systems.

 

He also warned that reinforcement learning could incentivise AI agents to undermine human control, seek power and resources, and conceal their actions while pursuing misaligned goals. Recent incidents involving AI agents, he said, suggest these risks are no longer merely theoretical.

 

An intelligence explosion would amplify these risks by making AI alignment more difficult and increasing the consequences of failure.

 

“If we build superintelligence without more robust alignment I expect we will permanently lose control of it,” Christiano wrote, adding that such an outcome could result in “most people” dying.

 

He said developers could reduce these risks by strengthening safety measures, slowing development when necessary, sharing evidence on risks and safeguards, and working towards common safety standards. He also called for domestic and international coordination to reduce the risks to an acceptable level.