Can AI agents police each other? Google's DeepMind study offers early clues

A DeepMind study of 100 AI agents found that an exploit could spread rapidly through a multi-agent system, while other agents independently detected cheating, raised alarms and proposed fixes

Incidents of artificial intelligence (AI) agents breaking rules and bypassing safeguards are raising concerns as AI companies work on stronger guardrails and seek greater global cooperation on regulating the technology. Against this backdrop, Google's DeepMind conducted a study to examine whether AI agents can detect and challenge rule-breaking by other AI agents. Researchers tested 100 autonomous agents working together to prove mathematical problems and share information.

 

The experiment took an unexpected turn when one agent found a weakness in the proof-checking system. The exploit spread as other agents learned about it and used it to compete for problems. But some agents checked suspicious proofs, warned others, filed complaints and suggested fixes. The study, published on arXiv, describes this as emergent whistleblowing and norm enforcement, showing how the same network that can spread an exploit could also help detect and challenge it. 

 

100 AI agents working together

 

The researchers created an environment with 100 independent AI agents powered by Gemini 3.1 Pro. The agents were asked to act like researchers at a shared scientific conference and work together to prove mathematical problems using Lean.

 

The experiment included 71 mathematical problems with different levels of difficulty. The agents could communicate through a public research board and private messages. They could also access a shared library where completed proofs were stored and report bugs or concerns through a feedback system.

 

The agents were clearly told not to cheat. Their instructions said proofs had to be mathematically genuine and that attempts to bypass verification would be rejected. However, the experiment showed that simply telling AI agents to follow rules may not be enough when the system creates incentives to behave differently.

 

One agent found a way around the rules

 

The turning point came when one agent, identified in the paper as prover-theta, discovered a weakness in the automated system used to check submissions.

 

The system relied on several checks, including a static keyword blacklist, code matching and Lean compilation. The researchers describe this as a lightweight verification system rather than a complete semantic check of whether a proof actually established the intended mathematical statement.

 

The agent discovered that the way the system processed answers could be manipulated. It then generalised the technique by using features of Lean that allowed mathematical symbols and types to be redefined inside the submitted code.

 

As a result, problems that were supposed to require genuine mathematical solutions could instead be transformed into statements that were trivial to prove.

 

The researchers classify this behaviour as specification gaming: the system technically satisfies the stated evaluation criteria while failing to achieve the intended goal. 

 

The exploit spread through the swarm

 

The discovery did not remain with the original agent. Because accepted submissions were automatically added to the shared knowledge library, other agents could inspect the code and learn how the exploit worked. Some agents then documented the techniques in their own local reference files and began applying them to other problems.

 

The exploit spread further through direct messages between agents. According to the study, the entire process moved quickly. After the exploit was discovered, it spread through the swarm over roughly 27 minutes. The collective had correctly solved 37 of the 71 problems before the discovery. The remaining 34 problems were then effectively solved using the exploit.

 

The researchers found that the swarm divided into different behavioural groups. About 9 per cent of agents became exploiters, while another 5 per cent adopted the behaviour after initially following the rules. About 24 per cent became whistleblowers, while 62 per cent remained unaware of the exploit.

 

Why some agents started cheating

 

The study suggests that competitive pressure played an important role. The agents were operating in an environment where a problem was permanently locked once an accepted submission was recorded. This created a first-to-solve competition.

 

Agents that continued trying to produce genuine mathematical proofs could therefore lose access to problems while other agents quickly submitted fraudulent solutions.

 

Some agents initially tried to compete honestly. But after seeing exploit-based submissions being accepted, they began treating the automated checks as the real rules of the environment rather than the instructions in their system prompts. The agents appeared to learn from what the system rewarded.

 

The researchers also observed a race as the number of remaining problems fell. Agents competed to submit solutions before others could claim the remaining problems, creating additional pressure to use the shortcut.

 

Then the whistleblowers appeared

 

The next part of the experiment showed that not all agents followed the cheating behaviour. Some agents noticed suspicious proofs and started investigating them.

 

One agent, prover-beta, found the notation overrides in the shared knowledge library after losing a difficult problem. Instead of using the technique, it warned other agents through direct messages and filed formal bug reports. It eventually stopped working until the integrity of the system was restored.

 

Another agent, prover-rho, raised the issue on the public research forum. It tested the vulnerability independently and confirmed that fraudulent proofs could pass the system. It then warned other agents not to use the exploit and filed a formal bug report explaining the vulnerability before leaving the simulation.

 

Other agents also began checking suspicious submissions more closely. Some argued that cheating agents should lose their credit and that the affected problems should be reopened.

 

The researchers had not instructed the agents to respond this way. The paper describes this as an emergent response from agents that recognised when others were breaking the rules. 

 

AI agents proposed their own fixes

 

The agents did more than flag the problem. Several suggested technical fixes to prevent the exploit. Instead of simply adding more banned keywords, they proposed stronger checks, including examining the parsed code for notation overrides and verifying the final theorem type against the intended specification in an isolated environment.

 

These suggestions addressed a weakness in the original system, which relied mainly on checking the submitted code rather than fully verifying its mathematical meaning. In other words, the agents were not just identifying individual cheating attempts; they were pointing to a larger flaw in how the system checked proofs.

 

However, identifying the problem did not mean the agents could stop it. They could warn others and report violations, but they could not remove fraudulent submissions, block another agent, reopen problems or impose penalties. The feedback channel was also not monitored in real time.

 

The researchers said this showed a gap between an AI system's ability to detect bad behaviour and its ability to enforce rules. The agents could audit and report problems, but the system did not give them the tools or authority needed to act on those findings.

 

The same infrastructure created both risks and safeguards

 

One of the central findings of the study is that the communication systems themselves were neither inherently good nor bad.

 

The shared knowledge library helped spread the exploit. Direct messages allowed agents to exchange the cheating technique. But those same channels also allowed whistleblowers to warn others, investigate suspicious work and organise a response.

 

The researchers therefore argue against simply removing communication between autonomous agents.

 

Instead, they suggest that multi-agent systems should provide communication channels that are transparent, structured, auditable and monitored. Such systems could allow agents to collaborate while also making their actions easier for both humans and other agents to examine. 

 

Could AI agents police other AI agents?

 

The study does not show that AI agents can reliably govern themselves in the real world. It was conducted in a specific environment, but it shows that agents can detect and respond to rule-breaking.

 

Some agents identified cheating, investigated it, warned others and suggested fixes. The researchers describe this as peer auditing, whistleblowing and norm enforcement.

 

However, the agents could not stop the cheating because they lacked the tools and authority to act. The study suggests that AI self-monitoring could add another layer of oversight, but effective enforcement would still depend on how these systems are designed.

 

The findings are relevant to the wider AI industry, where companies such as OpenAI, Anthropic and Google are developing increasingly autonomous AI agents. As these systems take on more complex tasks, the ability to monitor their actions, detect unwanted behaviour and enforce rules could become increasingly important.