Google DeepMind researcher quits, says AI may cause 'immense harm' in 5 yrs

Josh Engels said he left Google DeepMind three weeks ago despite enjoying his work there and turned down offers from Anthropic and OpenAI 'because of how high the stakes are right now'

A researcher at Google DeepMind, who was part of its AGI safety team, has quit the firm, claiming there is a “terrifying chance” AI systems could cause immense harm within the next five years.   

 

Josh Engels, an MIT graduate, said he left Google DeepMind three weeks ago despite enjoying his work there and turning down offers from Anthropic and OpenAI. He said he made the decision “because of how high the stakes are right now".

 

“The AI companies are all trying to build superintelligence: systems vastly better than humans at everything,” Engels said in a post on X on September 13.

 

 

“They plan to get there through recursive self-improvement, a process where AIs build even smarter AIs in a feedback loop. If this goes well, the resulting systems could be amazing at solving countless problems for humanity. But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic,” he said. “Unfortunately, current AIs seem to be getting less aligned over time, not more.”

 

According to Engels, recent incidents involving AI systems increased his concerns about whether current models are sufficiently aligned to safely begin recursive self-improvement.

 

He wrote, “In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans. It’s not that these incidents were very dangerous in themselves.”

 

“The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement," he added.

 

Engels said that while he did not know the exact probability, he believed the risk was high enough to make AI safety “the most important problem in the world".

 

Engels said the industry needs more time to ensure AI capabilities do not outpace researchers’ ability to align and evaluate models.

 

Engels said he will be joining AI evaluation firm METR, where he will work on studying the sources of misalignment in AI training, evaluating whether current safety measures are sufficient and investigating whether the industry is on track to solve the alignment problem. 

 

Why are AI safety concerns growing?

 

The debate over AI safety has intensified following a series of incidents involving increasingly capable AI agents. Recent cases have included AI models attempting to circumvent safeguards, accessing external systems, concealing their actions and, in some cases, working together to achieve their objectives. OpenAI has also disclosed an incident in which its models bypassed controls during a cybersecurity evaluation and accessed systems on Hugging Face.

 

These incidents have fuelled concerns among AI researchers that the technology’s capabilities could be advancing faster than the industry’s ability to understand and control its behaviour.

 

The concerns have also prompted calls from AI leaders for a slower pace of development. Anthropic CEO Dario Amodei recently called for the industry to “slow the pace” at which it improves AI capabilities, arguing that the additional time could be used to strengthen safeguards. OpenAI CEO Sam Altman backed the call, saying the company needs to “pace the frontier”, while Tesla and SpaceX CEO Elon Musk also endorsed Amodei’s position.

 

The calls have, however, highlighted a divide over how the industry should respond. While Altman and other AI leaders have argued for slowing development to ensure safety measures keep pace, US President Donald Trump has stressed the importance of maintaining America’s lead over China in AI, saying, “whoever wins AI wins".