Explained: AI jailbreak - what is it, how does it work, why is it a concern

AI jailbreaks can bypass safety restrictions in AI models through crafted prompts, raising concerns about their potential misuse for cybercrime, data theft, and unauthorised access to sensitive system

The Frontier Model Forum (FMF), a US-based entity formed to promote the safe and responsible development of advanced artificial intelligence systems, has released the first version of a framework to assess the risks posed by AI jailbreaks. Its members include OpenAI, Microsoft, Google DeepMind and Anthropic. The framework aims to establish a common method for measuring the severity of attempts to bypass AI models' cybersecurity safeguards and help developers identify vulnerabilities that need attention.

 

This comes as AI models are increasingly being used for tasks beyond answering questions, including writing code, analysing data and accessing internal systems. While these models are designed to refuse harmful requests, carefully worded prompts can sometimes trick them into ignoring their safety restrictions, resulting in jailbreak.

What is an AI jailbreak

An AI jailbreak is an attempt to make an AI model ignore the safety rules set by its developers. These rules, often called guardrails, are designed to prevent models from generating harmful content or following instructions that could be used for illegal or dangerous activities.

 

For instance, an AI assistant may refuse to provide instructions for developing malware or breaking into a computer system. A jailbreak attempts to get around that refusal by changing the way the request is presented, breaking it into smaller steps or asking the model to respond in a particular role.

 

According to Microsoft Threat Intelligence, a jailbreak is a technique that causes an AI system's safeguards to fail. The consequences depend on what those safeguards were designed to prevent.

 

Two examples are DAN (Do Anything Now), which attempts to get ChatGPT to ignore its safety rules, and Crescendo, a technique identified by Microsoft that uses a series of harmless questions to steer an AI model towards restricted content. Microsoft said Crescendo could bypass safety filters in fewer than 10 exchanges.

How does an AI jailbreak work

AI models are trained to respond to instructions and generate useful answers. At the same time, developers add safety training and filters to stop them from responding to certain requests. A jailbreak tries to exploit gaps between these two objectives: the model's tendency to follow instructions and its obligation to refuse harmful ones.

 

One approach is to disguise a restricted request as a harmless task. For example, a user may ask the model to act as a fictional character, researcher or cybersecurity expert who is not bound by the usual restrictions. The aim is to persuade the model to treat the request as part of a role-play rather than an instruction that should be refused.

 

Another method involves hiding the intent of a request. This could mean using coded language, unusual spellings, translations or other forms of obfuscation. The model may still understand the underlying request even if it does not recognise it as a prohibited instruction.

 

Jailbreak attempts can also take place over several exchanges. Instead of asking for restricted information directly, a user may begin with ordinary questions and gradually move towards a harmful objective. Microsoft's 2024 research described a technique called Crescendo, in which a series of seemingly harmless prompts gradually steers a model towards producing content it would normally refuse.

 

Other methods include filling a model's context with fabricated examples in which an AI assistant appears to comply with harmful requests. Some attacks use specially crafted strings that can confuse a model's safety training. These techniques differ, but their purpose is the same that is to get the model to respond in a way its safeguards are meant to prevent.

Why are AI jailbreaks a concern

The risk depends on what an attacker can get the model to do after bypassing its safeguards. In a basic chatbot, the immediate outcome may be the generation of harmful or restricted content. The consequences can be more serious when the model is connected to company systems, databases, software or other tools.

 

Microsoft lists several possible risks, including unauthorised access to information, the theft of sensitive data, the generation of ransomware and attempts to bypass an organisation's security or compliance policies. A manipulated AI system could also produce instructions for criminal activity or influence decisions made by automated systems.

 

The concern is particularly relevant as companies deploy AI agents that can perform tasks with limited human intervention. An agent may be given access to business documents, customer records or software tools so that it can complete a task. If an attacker manipulates the model and the system does not have adequate restrictions on what the agent can access or execute, the consequences may extend beyond an inappropriate answer.

 

A successful jailbreak does not automatically mean that a company has been hacked or that its data has been stolen. The impact depends on the model's capabilities, the access it has been given and the safeguards around the wider application. Microsoft says the severity of an incident should be judged by its consequences, such as whether it results in unauthorised access or triggers automated actions, rather than by the jailbreak technique alone.

Why are AI models vulnerable to jailbreaks

One reason is that AI models are built to understand and follow instructions. They are trained on large amounts of information and are designed to respond to a wide range of questions. Safety training is added to restrict certain responses, but it may not cover every possible way in which a request can be framed.

 

According to Microsoft, language models can be influenced by the way instructions are presented. They may also produce different responses to similar inputs because their outputs are not always predictable. A prompt that fails to bypass a safeguard once may produce a different result when modified or repeated.

 

US-based AI company Ultralytics identifies a conflict between two objectives as one reason jailbreaks can work: models are trained to be helpful and follow instructions, but are also expected to refuse harmful requests. Attackers try to exploit this tension by presenting a restricted request in a form that the model interprets as acceptable.

 

This does not mean every attempt will succeed. Developers regularly test models for such weaknesses and update their safeguards as new techniques emerge. However, the methods used to bypass restrictions also change, making it difficult to rely on a single filter or safety measure. 

How does the FMF framework assess jailbreak risks

The framework released by the Frontier Model Forum is intended to give AI developers and security researchers a common way to assess successful jailbreaks. It does not treat every bypass as equally dangerous. Instead, it looks at what a jailbreak allows a model to do and how easily that capability can be used.

 

The framework uses four factors:

• Capability gain: What new or restricted capability does the jailbreak unlock?
• Capability scope: How broad or powerful is the capability that becomes available?
• Ease of deployment: How easily can someone use the jailbreak in practice?
• Ease of discovery: How easy is it for others to find and reproduce the technique?

The framework uses these scores to rank the risks posed by successful cyber jailbreaks, helping organisations identify the most serious vulnerabilities and prioritise fixes.

 

The FMF said Version 1.0 is an initial release and will be tested against different jailbreaks. It has invited public feedback until December 21, 2026, as it works towards refining the framework.

 

AI jailbreaks are not new, but their potential impact is changing as models gain more advanced capabilities and are connected to tools used by businesses. 

How can companies reduce the risk

There is no single measure to prevent every jailbreak. Microsoft recommends a zero-trust approach, under which organisations assume that AI models could be manipulated and put multiple safeguards in place. These include screening prompts and responses, managing user identities, restricting access to sensitive data and monitoring AI activity.

 

Companies should also limit the actions AI agents can perform and require human approval for sensitive tasks. Another measure is AI red teaming, in which security teams test models with different prompts and attack techniques to identify weaknesses before they can be exploited.

 

AI developers are also working on additional safeguards. Anthropic's Constitutional Classifiers examine user prompts and AI-generated responses to identify restricted content. In tests involving 10,000 jailbreak prompts, the company reported that the attack success rate fell from 86 per cent to 4.4 per cent. OpenAI, meanwhile, is developing activation classifiers that monitor a model's internal signals as it generates responses. The system can interrupt potentially harmful responses and block tool calls, adding another layer of protection against jailbreak attempts.