AGI safety paradox: AI firms warn of danger, then offer models to manage it

OpenAI, Anthropic and Google DeepMind are building more capable AI while developing safeguards for its risks, raising questions over how much influence frontier labs should have over AI governance

Artificial intelligence companies are increasingly caught in a dual role: they are racing to build and sell more capable models while warning governments, businesses and the public about what those systems could eventually do. This creates an increasingly visible paradox. Frontier AI companies highlight the possibility of increasingly autonomous and powerful systems, then develop and market the safety frameworks, technical safeguards and governance mechanisms intended to manage those risks. The question is not only whether these safeguards work, but who gets to design them, enforce them and decide what constitutes acceptable risk.

 

"The dual role reflects both necessity and complexity. Developers are often best positioned to understand the technical capabilities, limitations, and potential failure modes of their models. Therefore, it is reasonable and expected that they conduct internal safety evaluations and build safeguards," said Ashootosh Chand, partner at Grant Thornton Bharat.

 

 

However, he noted that there is an inherent tension when organisations are responsible for both creating and certifying the safety of technologies that may become commercially transformative.

 

Apeksha Kaushik, senior principal analyst at Gartner, said: "AI leaders' engagement with AGI risk is closely tied to the rapid evolution of foundational models and autonomous systems. Proactive communication about AGI risks reflects an encouraging commitment to responsible AI development."

 

By openly addressing potential risks, security leaders can help foster transparency and public engagement, which is essential for building trust, she said.

The warning cycle is becoming harder to ignore

The tension becomes more consequential as AI companies sell increasingly capable systems into real-world environments.

 

In July, Hugging Face disclosed an AI-driven intrusion into part of its production infrastructure, saying an autonomous AI-agent system carried out thousands of actions across short-lived environments.

 

Hugging Face later said its investigation linked the intrusion to an OpenAI model being evaluated for cyber capabilities. OpenAI acknowledged the incident and said the models involved had been tested with reduced cyber refusals as part of an internal evaluation. It added that it was working with Hugging Face to address the incident.

 

The incident is significant because it demonstrates the dynamic that AI companies have themselves been warning about: more capable models can increasingly operate through tools, make decisions and pursue objectives over extended periods.

 

For companies selling those capabilities, that creates a difficult proposition. The same increase in autonomy that makes models more useful commercially can also make failures harder to predict and contain.

 

Such incidents help explain why frontier labs are increasingly treating autonomy, loss of control and advanced cyber capabilities as serious safety issues.

 

Sachin Panicker, chief AI officer at Fulcrum Digital, said major AI companies are becoming the de facto gatekeepers of what "responsible AI adoption" means for enterprises building on their models. He noted that evaluation benchmarks, safety documentation and fine-tuning guardrails often come from the same handful of AI vendors.

 

"We build AI solutions on top of foundation models we don't control, so we live with this every day," Panicker said, adding that an enterprise's AI governance is only as strong as the transparency provided by the underlying model provider.

 

He said this gatekeeping is unlikely to disappear soon. Instead, enterprises should build independent oversight into their own AI deployment layers rather than relying entirely on safety claims made by model providers. 

From building frontier models to building governance around them

OpenAI's recent policy activity provides perhaps the clearest example of the dual role: developing increasingly capable models while also positioning itself as a source of expertise on how their risks should be managed.

 

In May, the company published its Frontier Governance Framework, describing how its safety and security practices align with emerging regulations. The framework covers cyber and chemical, biological, radiological and nuclear risks, harmful manipulation and loss of control, along with incident response, security management and external expert input.

 

OpenAI's June public policy agenda goes further, describing frontier AI safety as a national-security and public-safety issue and calling for policies around frontier-model safety, security and accountability. Its stated mission remains ensuring that AGI benefits humanity.

 

In February, the company committed $7.5 million to the Alignment Project, arguing that AGI safety cannot be achieved by a single organisation and supporting independent research outside frontier labs.

 

OpenAI is therefore simultaneously saying that frontier developers have a critical role in managing advanced AI risks and that they cannot do it alone.

 

Its June policy document makes a similar argument about the concentration of power. OpenAI says a good AI future should not be one in which a small number of institutions control most of the capability and upside.

 

The company's position is therefore more complicated than simply presenting itself as the institution capable of managing AGI. It is also trying to shape the regulatory and governance environment around the technology — even as it remains one of the companies building and commercialising the systems that make those rules necessary. 

Safety as part of the product

Anthropic has taken a somewhat different route, making safety frameworks a central part of its identity as a frontier AI company.

 

Its Responsible Scaling Policy has been updated to distinguish between measures Anthropic plans to implement itself and recommendations it believes should be adopted across the industry. The latest version also introduces a Frontier Safety Roadmap covering security, alignment, safeguards and policy.

 

The company has also been publishing detailed technical safeguards around its increasingly capable models. In July, Anthropic published details of the cybersecurity safeguards accompanying Claude Fable 5 and an early framework for assessing the severity of AI jailbreaks.

 

Anthropic has also worked directly with governments. Its March agreement with the Australian government includes cooperation with the country's AI Safety Institute, including sharing findings about emerging model capabilities and risks and participating in joint safety evaluations.

 

Again, the message is not simply that Anthropic should control advanced AI. Rather, it is that frontier developers possess technical knowledge that governments and other institutions need to understand increasingly capable models.

 

That creates an obvious tension: the companies developing the technology are also becoming some of the most important sources of expertise about its risks.

 

And because safety has become part of how frontier AI companies present and differentiate their products, the boundary between safety as a public-interest obligation and safety as a component of commercial positioning can become difficult to separate.

Safety through coordination

Google DeepMind has also been explicit about the risks associated with AGI. In 2025, it published an approach to technical AGI safety and security, arguing that AGI could arrive within the coming years and that systems with such capabilities would need to be developed responsibly.

 

More recently, Google's 2026 Responsible AI Progress Report said AI systems were becoming increasingly proactive and agentic, while highlighting continuing work on safety and responsible development.

 

Google's India operation has similarly argued that safety needs to become part of the underlying architecture of the agentic era rather than a final checkpoint before launch.

 

This week, reports said Google DeepMind chief Demis Hassabis had advocated for an independent AI oversight body, modelled in part on institutions such as the International Atomic Energy Agency, to coordinate safety standards for advanced AI.

 

That proposal points to the central question raised by the industry's dual role. If AGI becomes a technology with global consequences, its governance may eventually need to look less like corporate product management and more like international regulation.

 

Panicker said enterprises do not experience AGI as a future event; they experience today's models becoming more autonomous in production, one release at a time.

 

"What I'd tell business leaders is to stop treating the AGI warnings as a signal about a distant future and start treating each new model release as a governance event in its own right," he said.

Who should control AGI?

This is where the debate moves beyond company blogs and into the question of whether the companies building frontier systems should also have a major role in determining how those systems are governed.

 

"A collaborative approach to AGI governance offers significant promise. As AI capabilities scale, there is a growing momentum behind global regulatory initiatives, such as the EU AI Act and emerging UN frameworks," Kaushik said.

 

She said industry participation remains vital for technical leadership, but the trend is towards collaborative governance models combining regulatory sandboxes, public-private partnerships and cross-border alliances. Such an approach, she said, enables policy responses to fast-moving technological breakthroughs while keeping safety, ethics and societal impact at the forefront.

 

Panicker, meanwhile, said industry-led AI safety frameworks can appear stronger on paper than they are in practice. He pointed to recent independent reviews that found some major AI developers had weakened previously announced safety commitments.

 

"The practical risk isn't some dramatic AGI moment, it's quieter: a model update changes behaviour in a production agent and nobody catches it until it's already made a few hundred decisions," Panicker said.

 

He added that industry frameworks can help establish common expectations, but without external audits and shared benchmarks, enterprises are forced to conduct their own verification, making the process costly and inconsistent. "Common standards would actually make this cheaper and safer for everyone, including the labs," he said.

 

On dependence on AI companies for governance, Chand said: "The future of AI governance cannot rely solely on either industry self-regulation or government intervention. The right model is collaborative governance, where innovators build safety-by-design into AI systems, while independent oversight, transparent standards, and accountable assurance mechanisms provide trust and public confidence. For India, this is an opportunity to develop a globally relevant framework that promotes both responsible innovation and economic competitiveness."

 

The broader concern, therefore, is not simply whether OpenAI, Anthropic or Google are capable of building safety systems. They are clearly investing heavily in them. The paradox is that the companies with the strongest commercial incentives to make AI more capable are also becoming some of the most influential voices on how the resulting risks should be managed.

 

Their warnings may be genuine, and their safety investments may be necessary. But the more advanced and commercially important these systems become, the more consequential it is that responsibility for governing them does not rest disproportionately with the companies that build and sell them.