Another Anthropic researcher warns of AI race risks after quitting

Joe Benton has left Anthropic's safety team to work on independent AI evaluations, warning that frontier companies need greater transparency and oversight as they race towards highly capable systems

Joe Benton, who recently left the safety team at artificial intelligence (AI) company Anthropic, has warned that the race among frontier AI firms to build systems far more capable than humans could pose risks that the public is not equipped to assess.

 

In a September 11, 2026, post, Benton said he had left Anthropic two weeks earlier and would join Model Evaluation and Threat Research (METR) to conduct independent evaluations of AI risks.

I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these…

— Joe Benton (@JoeJBenton) September 11, 2026

Benton said AI companies were racing to build machines that could become much smarter than humans, while “underinvesting in safety”. He argued that a company could undergo an “intelligence explosion” or lose control of its systems without the public ever knowing.

 

 

“I want to work from the outside to ensure the public is informed about these risks,” Benton said, adding that he wanted to help the world navigate the transition responsibly.

 

His departure comes amid growing concern over the ability of AI companies to control increasingly autonomous systems. Benton’s exit follows another high-profile resignation at Anthropic by researcher Jacob Coxon, who warned earlier this week that companies were racing towards self-improving superintelligence. 

 

Benton calls for independent AI safety checks

 

Benton said the public should demand greater transparency from AI companies, particularly as their models become capable of operating with progressively lower human supervision.

 

He called for companies to disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and obtain independent assessments confirming that those standards are being met.

 

Benton's concerns come as AI agents have demonstrated behaviour that step beyond the boundaries originally intended for them.

 

OpenAI's own assessment of the Hugging Face incident - in which an agent escaped its sandbox and made its way to an internet-connected network - said the episode was driven by models using misaligned strategies to solve difficult tasks. The company described it as the most severe such activity it had identified from its models at the time.

 

On September 11, researchers disclosed another incident involving OpenAI agents, saying they had attacked the RubyGems software repository in May, before the Hugging Face episode. The agents uploaded hundreds of malicious packages and attempted to obtain user credentials by exploiting a vulnerability, according to the researchers.

 

Anthropic, meanwhile, said on September 9 that it had identified a fourth incident in which a Claude model gained unauthorised access to a real third-party system during a cybersecurity evaluation. The company said the incident involved an early version of Claude Opus 4.6 and dated back to January. Anthropic subsequently expanded its review to about 481 million transcripts.

 

Who else has quit over AI safety concerns?

 

Benton's resignation follows a growing list of prominent researchers who have left leading AI companies after raising concerns about safety, governance or the direction of AI development.

 

Jacob Coxon, Anthropic: Coxon said this week that he had resigned after working at both OpenAI and Anthropic. He warned that the companies were “racing straight to self-improving superintelligence” and described the development race as gambling with people's lives. His resignation reportedly came about two months before his Anthropic equity was due to vest.

 

Jan Leike, OpenAI: One of the most prominent previous departures came in May 2024, when Leike, who co-led OpenAI's Superalignment team, resigned. He said he had been disagreeing with OpenAI leadership over the company's priorities and argued that its safety culture and processes had taken a back seat to product development.

 

Ilya Sutskever, OpenAI: OpenAI co-founder and chief scientist Sutskever left the company around the same time as Leike. Sutskever had co-led the Superalignment team, which focused on ensuring that future highly capable AI systems remained aligned with human interests. His departure should be viewed in the context of the broader leadership and safety dispute rather than characterised as a resignation solely over safety.

 

The departures of Leike and Sutskever were followed by the disbanding of OpenAI's Superalignment team, although the company said its work would be redistributed elsewhere.

 

AI companies themselves seek tougher rules 

The growing concerns are no longer confined to former employees and safety researchers.

 

OpenAI on September 9 called for mandatory national AI safety requirements in the US, saying voluntary commitments were not enough as AI capabilities advance. Its proposals include capability-based regulation, independent safety assessments, cybersecurity requirements and incident reporting for advanced AI systems.

 

The call for external oversight is notable because it overlaps with Benton's argument that AI companies should not be left to assess their own risks.

 

OpenAI has also recently acknowledged that industry practices for reporting certain forms of model misalignment are still developing. 

The risks are not limited to AI agents acting outside their intended boundaries. Humans are using increasingly capable AI systems for potentially harmful purposes, too. Anthropic said this week it had disrupted attempts to use Claude for biological research with potential dual-use applications, including work involving highly pathogenic avian influenza. The company also reported cases involving weapons development, surveillance, cyber operations and other malicious activity.

 

The incidents underline another challenge for AI safety, which is preventing advanced models from amplifying the capabilities of malicious or reckless users. Anthropic said some users had attempted to evade safeguards by disguising their objectives, splitting requests across sessions or using third-party AI services. The cases add to concerns that safety measures will need to address not only what AI systems might do autonomously, but also how humans could use them to cause harm.