AI training opt out: How a new standard can help publishers protect content

A proposed policy token aims to simplify how publishers opt out of AI training. While companies such as OpenAI, Anthropic and Google have backed the proposal, it is yet to become a standard

Artificial intelligence (AI) companies use content available on the internet, including news articles, photographs, research papers and other material, to train their AI models. Digital publishers have challenged this practice, specifically the use of their content for model training without permission or compensation.

 

To date, there have been several ways to stop automated programmes from accessing websites, but these methods have limitations. Some require website owners to set separate restrictions for different AI companies, while others depend on those companies following the instructions.

 

Now, a proposed standard called the policy token aims to address this problem. It would give publishers a common way to tell AI companies whether their content can be used for training, without having to create separate instructions for every company.

 

 

The proposal has received support from several major technology companies, including OpenAI, Google, Microsoft and Anthropic. However, their timelines for adopting it differ.

What is the policy token and how will it work

The policy token is a set of instructions that publishers can add to a file called robots.txt, which is already used by websites to tell automated programmes which parts of their content they can access.

 

These automated programmes, known as web crawlers or bots, visit websites and collect information. Search engines such as Google use crawlers to discover new webpages and include them in search results. AI companies also use them to collect online material that may be used to train their models.

 

The policy token is being developed through the AI Preferences (AIPREF) working group of the Internet Engineering Task Force (IETF), an organisation that develops technical rules and standards used across the internet.

The proposal was submitted by Kevin Kelley, Anthropic's technical lead for standards. It introduces a shared identifier for AI training, which participating companies can recognise when their crawlers visit a website.

 

Under the proposal, a publisher could use this identifier to tell AI training crawlers not to access any part of its website. It could also make exceptions for selected sections. For instance, a news organisation could restrict access to its articles while allowing crawlers to access a designated press or media section.

 

The main purpose is to reduce the number of separate instructions publishers need to maintain. At present, a publisher wanting to prevent OpenAI from collecting its content for training must identify OpenAI's crawler in its robots.txt file. It must do the same for crawlers operated by Google, Anthropic and other companies.

 

With a shared identifier, publishers would be able to communicate the same restriction through one instruction. AI companies that adopt the proposal would be expected to recognise and respect it.

 

According to The Verge, the proposal has received support from four major AI companies, although their implementation plans differ. Anthropic has already adopted the signal, while OpenAI has said it plans to implement it in the coming months. Google has announced plans to support it in 2027. Microsoft has also committed to supporting the shared preference, with implementation planned from February 1, 2027.

Why do publishers need a common opt-out mechanism

The use of online content for AI training has become a point of disagreement between publishers and technology companies. AI developers collect large amounts of text, images and other material from websites to train their models. Publishers argue that the availability of their work on the internet should not automatically give technology companies permission to use it for commercial AI products.

 

There is also a business concern. AI-powered search services can use information from news articles and other websites to answer users' questions directly. This means readers may get the information they need without visiting the original website. For publishers that depend on website traffic and advertising revenue, this could affect their business.

 

However, publishers also need their content to remain discoverable through search engines and AI services. Blocking every automated programme could prevent their articles from appearing in search results or being referenced by AI-powered services.

 

This is why distinguishing between different uses of content matters.

 

For example, OpenAI operates two separate crawlers. GPTBot collects website content that may be used to train its AI models, while OAI-SearchBot helps ChatGPT find and display information from websites in search results. A publisher can block GPTBot while allowing OAI-SearchBot to access its articles.

 

The policy token is intended to make this distinction easier to manage by providing a shared instruction specifically for AI training.

What are the existing ways to prevent AI companies from using website content

Publishers already have several ways to restrict how AI companies access their content, but these approaches have different limitations.

 

Robots.txt is the most common method. It is a file through which website owners tell automated programmes which parts of their websites they can access. Publishers can use it to block individual crawlers, such as OpenAI's GPTBot. However, they must create separate instructions for different companies. More importantly, robots.txt is a request rather than a technical barrier. A crawler can ignore it and still access publicly available content.

 

Some companies have introduced their own controls. Google, for instance, offers Google-Extended, which allows publishers to restrict the use of their content for Gemini training without affecting their visibility in Google Search. OpenAI also separates its training crawler, GPTBot, from OAI-SearchBot, which helps ChatGPT find information for search results. While these controls offer more choice, publishers still have to manage different settings for different platforms.

 

Cloudflare, a company that provides website security and traffic management services, offers another approach through its Content Signals system, which lets publishers allow content to be used for search while restricting its use for AI training. Its Pay Per Crawl service takes this further by allowing publishers to charge AI companies for accessing their content. However, both approaches depend on AI companies recognising the instructions or agreeing to pay for access.

There are also proposals focused on licensing content. Really Simple Licensing (RSL), an open standard published in December 2025, allows publishers to specify conditions for using their material, including whether AI training is permitted, attribution is required or payment must be made. It is supported by organisations including Cloudflare, Yahoo, Vox Media and O'Reilly Media. However, it too depends on AI companies recognising and following the declared terms.

 

Other projects attempt to make content harder to extract. ShieldFont, developed by France-based S&A Studio with Danish type foundry Playtype, changes the text stored in a webpage while displaying the intended words to human readers. A basic scraper could therefore collect altered words instead of the original text. However, more advanced tools that capture webpage images and use optical character recognition (OCR) can potentially recover the displayed text. The technique can also create accessibility and search-related problems if not implemented carefully.

 

Another option is the Text and Data Mining Reservation Protocol (TDMRep), which allows publishers to declare whether their content can be used for activities such as AI training and data mining. The International Press Telecommunications Council (IPTC) also recommends embedding copyright information in images and videos to identify their creators and communicate usage restrictions.

How could the policy token address these limitations

The policy token aims to reduce the effort involved in managing restrictions across different AI companies. A publisher would not have to create a separate training restriction for every crawler, provided the companies collecting its content support the shared signal.

 

It could also help publishers distinguish between training and other uses of their content. A news website, for instance, could restrict its articles from being collected for training while allowing search crawlers to index them. This would help it remain visible in search results.

 

However, the proposal has an important limitation. It is an instruction, not a technical barrier. It cannot physically stop a crawler from accessing a website.

 

Even if major AI companies agree to follow the signal, it cannot prevent unknown bots, companies that have not adopted the standard or operators that deliberately ignore website instructions from collecting publicly accessible content. Publishers may still need website security services, bot-blocking tools and legal action to protect their work.

 

There is also the question of content that AI companies have already collected. Adding a new instruction to robots.txt cannot automatically remove articles from existing training datasets or reverse their use in models that have already been trained. The proposal mainly concerns how participating crawlers handle future access to websites.

 

The policy token is also still an Internet-Draft, which means it is a proposal under discussion and has not yet become a formally adopted internet standard. Its scope is limited to communicating AI training preferences. It does not provide a complete system for licensing content or collecting payments from AI companies.