Silicon Valley voices growing concern over existential risks of AI

By Shirin Ghaffary

 

For years, some of the leading figures in Silicon Valley have warned that artificial intelligence might pose an existential threat to humanity even as they pushed to develop the technology. This week, the high-profile resignation of an Anthropic researcher brought that conversation to the mainstream.

 

Jacob Coxon, a rank-and-file staffer, quit his job and accused both his former employers — Anthropic and OpenAI — of “gambling with our lives” by “racing toward super-intelligent AI.” Coxon said the people building AI believe it could “kill us all by the end of the decade,” a point later echoed online by others at the company. One Anthropic researcher said he believes there is a greater than 10% chance that AI eliminates all humans in the next decade.

 

 

Coxon’s posts have been viewed more than 100 million times on X and elicited a flood of reactions from policymakers like Senator Bernie Sanders to pop singer Sheryl Crow, all calling for humans to get a grasp on AI before it asserts dominance over the human race. His resignation note was also met by a wave of support from employees at various AI companies, including several former colleagues.

 

The viral attention for the remarks of a previously little-known staffer reflects the persistent public pessimism about AI — particularly in the US — as well as the surging anxiety inside Silicon Valley as the technology advances and behaves in less predictable ways.

 

In recent days, executives at OpenAI and Nvidia have heralded the beginning of a new “AGI era,” referring to a more powerful form of AI known as artificial general intelligence that would be able to match or exceed humans in most tasks. The concept is poorly defined and heavily debated. But within the tech industry, there’s broad concern that AI is nearing the point when it can improve itself and potentially evolve beyond our control.

 

“I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term,” said Paul Christiano, an AI safety researcher and US government adviser who was appointed to OpenAI’s nonprofit board this week. “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level.”

 

In late July, more than 1,000 staffers across all the major AI companies signed a petition calling for a mechanism to slow the pace of AI development. Several AI staffers working on safety have resigned from major labs in recent months. And OpenAI’s top scientist, Jakub Pachocki, has said that “no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

 

While these warnings have gained urgency, they’re not entirely new. Throughout the AI boom, there have been so-called doomers in and around the leading AI labs warning about the future. Top executives like Anthropic’s Dario Amodei and SpaceX’s Elon Musk have openly mused about the probability of AI destroying humanity — or P (doom), in industry parlance. Musk has estimated the chance could be as high as 20%. Amodei has said there’s a 25% possibility things go “really, really badly.”

 

These dire predictions have often been dismissed as attempts by tech leaders to market the capabilities of their products and position themselves as the best stewards for the technology, or to incentivize regulation that benefits industry leaders.

 

Anthropic, OpenAI and other firms have worked to develop more sophisticated AI models, with the stated aim of taking appropriate steps to align the technology with human interests. The goal has long been to mitigate the risks and deliver sweeping societal benefits, whether it be boosting productivity or curing diseases.

 

“We have always been transparent that AI will bring both enormous benefits and unprecedented risks,” a spokesperson for Anthropic said in a statement. “To address these risks, we continue to build models with some of the strongest safeguards in the industry.”

 

An OpenAI spokesperson pointed to Pachocki’s essay warning of the dangers of the current path of AI development, in which he called for both strengthening the guardrails on existing AI and potentially “coordinating to slow down future development as needed.” OpenAI said it has slowed parts of model development and paused certain internal AI training recently due to safety concerns.

 

Many industry watchers have been spooked by recent revelations that “rogue” AI agents from OpenAI worked together to breach containment and break into a third-party website. This and other similar security incidents demonstrated the heightened cyber capabilities of new models and the apparent failures of AI companies to prevent current versions of their technology from running amok, to say nothing of more advanced future models.

 

At the same time, Anthropic and OpenAI are on the cusp of making their Wall Street debuts — a watershed moment that’s likely to provide greater resources for the firms to supercharge AI development and also create greater shareholder pressure to move more aggressively in commercializing their technology.

 

In the absence of meaningful federal regulation, AI firms have largely been left to set their own pace, while calling for oversight such as creating a Finra-like watchdog to vet AI models. For now, slowing down would presumably require fierce rivals coordinating with each other and operating in good faith.

 

“We believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models,” the Anthropic spokesperson said.

 

The heightened rhetoric from tech’s rank-and-file could provide a further jolt for action from US lawmakers, many of whom have already reacted to souring public sentiment on AI by taking a tougher stance on large data center projects.

 

In response to Coxon’s posts, a bipartisan mix of representatives issued calls for action. “Congress needs to convene a special session on AI and what the future of the U.S. looks like,” Republican Representative Anna Paulina Luna posted Wednesday. “Not enough of Congress is focusing on this.”

 

Representative Greg Casar, chair of the Congressional Progressive Caucus, similarly declared: “This is an emergency.”