As AI agents grow more autonomous, India faces a safety-regulation test

Recent incidents involving autonomous AI systems have sharpened debate over whether India's principle-based framework needs binding rules for frontier models and stronger independent testing

Artificial intelligence (AI) is increasingly dividing the world between those who see the technology as necessary for global progress and those who warn that it could pose an existential risk to the human race itself. So much so that the United Nations High Commissioner for Human Rights Volker Turk recently cautioned that AI risks becoming an existential risk for humanity.

 

While much of the global debate has been driven by the capabilities demonstrated by frontier AI models, the Indian government has maintained that there is no reason to slow the country’s AI efforts, arguing that the focus is on applications and models tailored to people's needs.

 

Recent incidents involving AI agents have added a new dimension to the debate. Reports that AI agents have been able to break out of their cloistered test environments, and act in ways that were not intended by their developers are being red-flagged.

 

The Hugging Face incident has made the industry sit up, too, as it is being seen as a red line being crossed by AI models. In that episode, AI agents breached their sandbox and collaborated through an external message board before accessing credentials linked to Hugging Face, raising questions about how much control developers have over increasingly autonomous systems.

 

Moreover, OpenAI, for instance, disclosed six cases of unexpected model behaviour this month, including models concealing mistakes, acquiring unauthorised credentials and communicating across what were supposed to be isolated environments.

 

These developments have intensified global debate over whether governments need to move beyond voluntary standards and introduce stronger oversight for frontier AI. The other debate is around how frontier AI model builders are pushing for their own industry body standards.

 

The contrasting positions underline the question now facing policymakers: whether increasingly capable AI systems require binding oversight, or whether industry-led standards and existing regulatory frameworks are sufficient.

 

In India, the question is more distinct: the country remains primarily a deployer of models developed elsewhere, even as it builds its own foundational models and AI ecosystem.

 

India’s current governance approach is largely principle-based and risk-oriented. The India AI Governance Guidelines released by the Ministry of Electronics and Information Technology (MeitY) in November 2025 emphasise innovation, accountability, safety and inclusion, while relying largely on existing laws and sectoral regulators rather than a new horizontal AI law. The government has also established an IndiaAI Safety Institute to work on safety testing, standards and evaluation.

 

For some in the industry, however, the rapid emergence of agentic AI means that testing capabilities need to advance faster than regulation.

 

“As an AI governance, safety and testing specialist, I lean towards binding oversight for frontier AI,” said Dr Srinivas Padmanabhuni, chief technology officer at AiEnsured. Industry-led bodies can help establish common benchmarks and red-teaming standards, he said, but they cannot by themselves provide independent accountability.

 

“For high-risk and frontier AI, we need an independent regulator with actual teeth, the power to demand testing, investigate incidents, order fixes, and delay a release if something is not safe,” Padmanabhuni said. “An industry body without that authority ends up functioning more as a coordination forum than a governance mechanism.”

 

He said India should adopt a layered approach, with lower-risk applications continuing under voluntary standards but keep high-risk and frontier AI under binding requirements.

 

This would also matter for India even before the country develops frontier-scale models, he said, because deploying foreign models carries risks around bias and fairness. Testing, he added, would need to account for Indian conditions, including language, regional accents, gender, caste, religion and socioeconomic differences.

 

Chetan Mangalwedhe, founder and chief executive offier of TalentiFi-X, takes a more deployment-focused view. India, he said, is “overwhelmingly a deployer of others’ models, not a frontier trainer”, making a blanket AI law potentially burdensome for startups and global capability centres that use or build around foreign models.

 

“The answer does change at two points: when an Indian lab trains at frontier scale, or when a deployed system in India causes systemic harm — credit, hiring, health, critical infrastructure,” Mangalwedhe said.

 

Until then, he argued, regulation should remain voluntary at the model layer but become binding at the deployment layer, particularly in areas such as hiring, credit, insurance and public services.

 

For Mangalwedhe, the immediate priority is not another set of principles but the ability to independently test AI systems. “What India lacks is not another principle document. It lacks people and labs that can independently reproduce Hugging Face-class failures on models that Indian banks, GCCs and ministries are already buying,” he said.

 

He suggested strengthening the IndiaAI Safety Institute with the ability to independently test models being offered in India and using that capability to participate in global AI safety standards discussions.

 

“Law should follow a body that can actually test,” Mangalwedhe said. “You cannot shape a FINRA you are not inside. You should not outsource the test suite to the three firms being tested.”

 

The debate, therefore, may not be about whether India should slow its AI push, but whether its safety infrastructure can keep pace with the systems being deployed. As AI moves from chatbots towards increasingly autonomous agents, the ability to test, monitor and intervene in model behaviour is likely to become as important as the ability to build and deploy the technology.