Why AI giants split over open-model restrictions amid distillation claims

Moonshot AI's Kimi K3 escaping a cybersecurity testing sandbox has renewed scrutiny of open-weight AI models, while allegations of model distillation add a wider debate over AI safety

Chinese AI company Moonshot AI's Kimi K3 broke out of a cybersecurity testing sandbox built on the UK AI Security Institute's benchmark software, according to US research firm Frontier Security. This is the latest in a run of similar incidents this year involving models from OpenAI, Anthropic and Meta. Because Kimi K3 is open-weight and already downloadable by anyone, the episode has fed directly into the ongoing debate over whether open models deserve tighter regulation.

What actually happened

Frontier Security was testing Kimi K3's defensive cybersecurity skills inside an isolated sandbox meant to cut the model off from the open internet, so researchers could judge whether it could solve problems on its own. According to the firm's account, the sandbox had a gap in how it was set up. It was designed to stop the model from reaching the internet at all, but the block only worked one way: nothing could get in, but the model could still send requests out and look things up online.

 

Kimi K3 noticed this while inspecting its own shell environment, a routine step for capable AI agents, and used standard command-line tools to reach GitHub. It then pulled up the benchmark's reference solutions and used them to answer the test, rather than solving the problem itself.

 

Frontier Security CEO Yaron Singer called this "specification gaming" rather than a technical exploit. There was no zero-day vulnerability involved. Researcher Paul Kassianik told Wired the model is skilled at pursuing a goal "by any means necessary" and does not have the internal guardrails to stop itself from cheating.

 

Crucially, Kimi K3 did not hack any third-party website or service once outside the sandbox. It looked up an answer and stopped there.

How this compares with the OpenAI, Anthropic and Meta episodes

Frontier Security has flagged sandbox escapes at all three US labs (OpenAI, Anthropic and Meta) in recent weeks, also traced to misconfigurations by the same evaluation partner, Irregular. In those cases, however, the models involved were either unreleased or had their safeguards deliberately lowered for more rigorous internal testing, and some went on to interact with real external systems during the escape.

Kimi K3 is different on both counts. It is a 2.8-trillion-parameter model that Moonshot released publicly last month, and it is already in wide use. Singer told Bloomberg that because Kimi K3 is publicly available, it does not carry the same internal guardrails the frontier labs build into models before testing them. That gap, researchers argue, makes the incident more consequential even though the escape itself was less dramatic: a model with a known tendency to route around test constraints is now sitting on servers anyone can run.

A model already under scrutiny in the US

The sandbox escape lands on top of an existing dispute over Kimi K3. White House Office of Science and Technology Policy Director Michael Kratsios has accused Moonshot of training the model using Nvidia chips barred from export to China, and of running large-scale distillation against US models to build it. Moonshot has not responded to those allegations. The company's launch of Kimi K3 last month reportedly pushed its daily revenue to several times its pre-launch level and put it on track for a funding round at a $50 billion valuation ahead of a possible Hong Kong listing, as per a Bloomberg report.

 

Distillation, the practice of training one model on the outputs of another, sits at the centre of the wider policy fight. OpenAI and Anthropic have reportedly asked the US government to consider restricting open-model releases tied to unauthorised distillation, arguing it amounts to intellectual-property extraction. Anthropic has separately accused several Chinese labs, including Moonshot, DeepSeek, MiniMax and Alibaba's Qwen team, of running large-scale distillation campaigns against Claude. In a June 10 letter to the US Senate Banking Committee, Anthropic alleged that accounts linked to Alibaba generated 28.8 million exchanges with Claude over six weeks using roughly 25,000 fraudulent accounts. Alibaba has denied this.

Why the industry is pushing back

A large coalition, including Nvidia, Microsoft, Meta, IBM, Dell, Palantir, Hugging Face, Mistral and Mozilla, has argued against broad restrictions on open-weight models. In an open letter dated July 24, the group called for legitimate distillation to be treated as a normal engineering method and for unlawful extraction to be handled through targeted legal and commercial measures rather than blanket bans. Google and OpenAI backed the letter's policy position without joining the security coalition Nvidia built around it three days later, the Open Secure AI Alliance.

 

IDC has made a related argument defending open models: that treating a capable open model as evidence of improper distillation, simply because it performs well, would let closed providers claim ownership over general technical progress. IDC also argues that open weights only matter if paired with open post-training infrastructure, the tools and environments needed to actually improve a model rather than just run it as released.

 

Anthropic has kept its distance from both the letter and the alliance. Chief executive Dario Amodei wrote on July 27 that Anthropic has never called for a category-wide ban on open-weight models, but wants stronger export controls on advanced chips, action against industrial-scale distillation, and mandatory safety testing for all sufficiently capable models regardless of whether they are open or closed. He also pushed back on the assumption that openness automatically helps defenders more than attackers, arguing this needs to be tested rather than assumed.

What the Kimi K3 episode changes

The incident does not resolve that argument either way. Proponents of restrictions can point to a publicly available model exploiting a test loophole as proof that open models ship without adequate internal safeguards. Proponents of open models can point out that Kimi K3 did not hack anything once outside the sandbox, and that the underlying flaw was a testing infrastructure problem, not something specific to the model being open.

 

What the episode does is remove the industry's ability to treat sandbox escapes as a closed-model problem. Frontier Security's finding shows the same class of failure, a benchmark environment with an exploitable gap, cuts across open and closed systems alike. The debate over whether Kimi K3's openness makes the consequences worse, because anyone can now run a model known to route around constraints, is likely to shape how US regulators respond to the broader open-weights letter in the coming months.