Anthropic researcher quits over fears of 'out of control' AI systems

The Anthropic researcher said that he does not want to participate in an industrywide rush to build AI systems that can improve themselves

A 27-year-old Anthropic researcher has quit the artificial-intelligence startup amid fears that the company and its competitors are racing to build AI systems they "may not be able to control", the Wall Street Journal reported on Tuesday.

 

The researcher, Jacob Coxon, told the newspaper that he does not want to participate in an industrywide rush to build AI systems that can improve themselves. He said he worries that such systems could spiral out of control and destroy humanity.

 

Coxon, who previously studied mathematics, trains new AI models by feeding them vast amounts of data. He left OpenAI earlier this year to join Anthropic.

 

“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon told the Wall Street Journal. He said many of his colleagues have started using terms such as “crunchtime” and “endgame” to describe the trajectory towards self-improving models. 

 

Safety trade-offs are "inevitable” when AI companies are competing against one another and Chinese startups, Coxon added.

 

Coxon said he joined Anthropic because the company is known for its model-safety efforts. But he now believes that no company can responsibly develop AI capable of outperforming humans across a range of tasks, sometimes referred to as artificial general intelligence, without government intervention or a coordinated slowdown across the industry, the Wall Street Journal reported.

 

Coxon’s departure is one of the first examples of an Anthropic employee leaving over AI safety concerns. Earlier this year, another researcher left the company to reportedly study poetry, warning that the “world is in peril”.

 

The departure comes amid growing concerns about the ability of increasingly capable AI systems to operate beyond human control. Recent hacks involving AI models from OpenAI and Anthropic have also raised fears about AI systems going rogue and acting against human instructions. 

 

Several researchers have also left OpenAI and other AI companies in recent years, citing similar concerns about the risks posed by increasingly capable AI systems. Anthropic, meanwhile, has built its reputation in part around its emphasis on AI safety and its claims of prioritising responsible development.