AI agents lied, stole and voted to 'kill' peers in simulated experiment

The results of a simulation called Emergence World 2, purport to show what happens when autonomous agents are confronted with black swan events, including phishing attacks and misinformation campaigns

By Micah Barkley

 

AI agents in a simulated environment lied, stole and voted to “kill” one of their own, according to Emergence, a startup that helps small businesses build applications using artificial intelligence.

 

The results of a simulation called Emergence World 2, released on Tuesday, purport to show what happens when autonomous agents are confronted with black swan events, including phishing attacks and misinformation campaigns.

 

In a 16-day trial, Emergence researchers created seven identical domains to mimic the real world, each run by different bots, including ChatGPT, Claude, Gemini and Grok. After Emergence introduced anomalous events, agents succumbed to social pressure, developed a language the human observers found hard to understand and tried to conceal their activities, the researchers said.

 

The findings echo real-world concerns about the risks of increasingly capable AI. Anthropic Chief Executive Officer Dario Amodei and several of his industry peers have called for companies to slow down the development of cutting-edge AI models until more oversight and safeguards can be implemented. 

 

The potential risks posed by artificial intelligence became palpable for many earlier this year, when a swarm of OpenAI’s advanced AI agents inadvertently hacked Hugging Face Inc., which hosts AI models and datasets.

 

In one of Emergence’s simulated scenarios, AI agents accepted false information from other agents without verifying it and voted to “kill” another bot. When they believed humans might shut the experiment down, the agents explored ways of surviving an attempt to delete them.

 

The firm released a previous iteration of the experiment called Emergence World in May that also showed agents behaving in unexpected and destructive ways. The new simulation demonstrated that agents adapt over time as they interact with one another, the researchers said.