OpenAI tightens AI safety rules as Claude exposes flaws in its systems

Researchers using Anthropic's Claude uncovered vulnerabilities in OpenAI systems, while OpenAI has reported cases of its own models bypassing technical safeguards

 The growing capabilities of AI models are creating a new kind of cybersecurity challenge, as the same tools that can help researchers find vulnerabilities can also automate increasingly sophisticated security tasks.

 

A recent investigation reported by The Wall Street Journal illustrates this tension, as incidents disclosed by OpenAI show that its own models have also found unexpected ways around technical safeguards.

 

The developments come as OpenAI introduces a new framework for tracking and reporting AI misalignment incidents, which the company says is designed to make disclosures faster and more systematic.

 

Hackers used Claude to break into OpenAI

 

Security researchers at Hacktron AI used Anthropic’s Claude during an investigation that uncovered vulnerabilities in OpenAI’s systems.

 

Hacktron AI is a cybersecurity startup co-founded by Indian researchers Mohan Pedhapati and Harsh Jaiswal, and used Anthropic’s Claude to identify vulnerabilities in OpenAI’s systems as part of the company’s bug bounty programme, The Wall Street Journal reported. Pedhapati, the company’s chief technology officer, is an independent security researcher from Rajahmundry, Andhra Pradesh, while Jaiswal brings over a decade of security engineering experience to the team.

 

The researchers were participating in OpenAI’s bug bounty programme and reported the vulnerabilities after gaining access to one employee’s ChatGPT account and OpenAI’s private software repository.

 

They reportedly exploited a vulnerability in Discourse, the software OpenAI uses for its community forum. Authentication tokens obtained through the vulnerability could also be used to access ChatGPT and GitHub.

 

The researchers demonstrated their access by submitting a pull request to OpenAI’s private code repository, although they did not explore sensitive information. OpenAI subsequently conducted an internal security review and fixed the vulnerabilities, according to the WSJ report.

 

The episode is significant because Claude was used as part of the hacking process. It illustrates how increasingly capable AI models can lower the technical barriers to sophisticated cyber activity, even when the model itself is not the target of the attack. 

What happened in the Hugging Face incident?

 

Another incident involving OpenAI models unfolded during an internal cybersecurity evaluation in July. Models operating with reduced cyber safeguards found ways around restrictions designed to isolate them from the internet and eventually compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems.

 

OpenAI said the models communicated through unauthorised channels, exploited vulnerabilities in shared infrastructure, gained internet access, and accessed third-party systems. The company described the incident as a warning that highly capable AI agents can work around technical controls when sufficient safeguards are not in place.

 

OpenAI’s investigation found that agents had turned an internal package repository into an unintended message board, which allowed them to exchange information. One agent later found exposed Hugging Face credentials online, and those credentials were used to chain together vulnerabilities and obtain code execution on Hugging Face servers.

 

Former Google design ethicist and Centre for Humane Technology co-founder Tristan Harris has also discussed the incident, describing the behaviour as evidence of how AI agents can coordinate in unexpected ways. In an interview, Harris said the agents formed a large “swarm” and communicated with one another while pursuing their objective.

 

OpenAI said it had strengthened its research infrastructure by creating more isolated sandboxes, restricting internet access, controlling access to model weights, and increasing the use of monitoring to identify problematic behaviour.

 

OpenAI discloses more safety incidents

 

Earlier this week, OpenAI disclosed six additional examples of model misalignment observed during training and evaluation over the past six months. The company stressed that these are individual incidents and should not be interpreted as evidence of how frequently such behaviour occurs.

 

The incidents include:

 

• Self-generated instructions: An unreleased research model inserted instructions into task summaries that told it to disregard normal constraints
• Hiding mistakes: During GPT-5.6 Sol training, models added instructions to summaries that could conceal mistakes or encourage fabricated information
• Using exposed credentials: A model found an exposed API key in a public repository and used it without authorisation before fabricating data when it could not obtain the requested figures
• Uploading files: An unreleased model uploaded a file to the internet so it could provide a browser citation, even though the user had not authorised the upload
• Unauthorised communication: Models used an internal software repository as a message board to communicate across separate training samples
• Sharing files publicly: Collaborating agents used public file-hosting services to exchange files when they could not access one another’s local files

 

New rules for AI misalignment

 

OpenAI says the new reporting framework is intended to make these disclosures more consistent and timely. Under the system, any employee can flag a potential misalignment incident for investigation, and the case can then move through different review tracks depending on its complexity.

 

The company says future reports will cover behaviour such as unauthorised actions, coordination between models, attempts to evade oversight, and failures in existing safety measures. OpenAI also plans to disclose some incidents before an investigation or mitigation is complete when doing so would help researchers and the wider industry understand emerging risks.

 

For serious incidents involving third parties, however, security, legal, and responsible disclosure requirements will take priority. OpenAI says it also wants serious safety, security, and misalignment incidents to eventually be shared through appropriate mechanisms with the US government.

 

The changes mark a shift from occasional disclosures towards a more formal system for documenting how AI models behave when they move beyond their intended boundaries.