On October 30, 2026, OpenAI announced that it will support a common mechanism that lets websites tell multiple AI companies whether their content may be used to train generative AI. The mechanism uses robots.txt, the file that controls web crawler behavior, to state a training-use preference, and OpenAI aims to implement it within the next few months. On the same day, Google set a goal of supporting it during 2027, and Microsoft announced that Bing will honor the new directive starting February 1, 2027.
It is important to distinguish between AI model "training," "grounding," and web page "crawling." Training refers to using content to update a model's parameters. Grounding refers to the mechanism by which AI references information on the web when generating an answer. Crawling is the process by which a crawler retrieves the content of a page, and opting out of training use does not necessarily prohibit the retrieval of the page itself.
Until now, site operators have had to identify crawlers from each company, such as OpenAI and Google, and manage individually whether to allow training use. As more companies support a common directive, the burden of managing these settings one by one should shrink. However, the handling of use in AI answer generation and the relationship to existing settings are not uniform, so each company's specifications need to be checked.
A shared robots.txt directive for AI training preferences
robots.txt is a file that site operators use to tell web crawlers which parts of a site they may access. The new common signal uses this mechanism to indicate whether web content may be used for foundational training of generative AI models. OpenAI says it will take part in a voluntary effort in which companies respect site operators' wishes, while also maintaining existing crawler management through robots.txt.
Microsoft has published the following example of a setting that opts out of AI training use.
User-Agent: __usage_ai-training
Disallow: /This __usage_ai-training is not a name that identifies a specific crawler. It is a policy token that conveys a usage policy for the purpose of AI training. Combined with Disallow: /, it lets a site declare that none of its content should be used for AI training. It is handled separately from crawl restrictions for ordinary crawlers such as Bingbot, so adding this directive alone does not stop access to web pages or crawling by search engines.
Discussions toward standardizing this mechanism are continuing in the IETF's AIPREF working group. Microsoft also says further work is needed before a final standard is established. The announcements do not guarantee that all AI companies will follow the common signal, nor do they make it possible to remove the influence of content from models that have already been trained.
OpenAI, Google and Microsoft differ in timing and conditions of use
Comparing the policies the three companies presented on October 30, 2026, they agree on advancing a common way to express AI training preferences, but differ in timing, in the relationship to existing settings, and in conditions for use in AI answers.
We summarized each company's approach based on OpenAI's announcement, Google's announcement, and Microsoft's announcement. Note that the following reflects planned implementations and operating policies at the time of the announcements, and does not mean all features are already available.
| Item | OpenAI | Microsoft / Bing | |
|---|---|---|---|
| Timing | Aims to implement within the next few months | Aims to support it during 2027 | Honors the new directive when crawling from February 1, 2027 |
| Relationship to existing settings | Maintains existing crawler management. Detailed order of precedence not published | Plans to treat the common training opt-out the same as a Google-Extended opt-out when Google-Extended is not explicitly specified. Individual specification remains available | For training, site/path-level specification via robots.txt and page-level specification can be used; page-level takes precedence |
| Use in AI answers | Details not specified in this announcement | Sites that explicitly allow Google-Extended can be eligible for grounding and clicks in Gemini Apps | A separate answer-use setting independent of training, selectable from three levels per page |
For Google, the relationship between the common signal and the existing Google-Extended is key. For sites that do not explicitly specify Google-Extended, Google plans to treat the common training opt-out the same as a Google-Extended opt-out. Where Google-Extended is specified individually, that setting remains usable. According to Google, sites that explicitly allow Google-Extended can be eligible for grounding and clicks by Gemini Apps.
Microsoft makes clear that it treats AI training and grounding as independent settings. Settings for AI answer generation can be specified per page, with a choice of three values: Allow, DisallowExtended, and Disallow. Of these, DisallowExtended permits use in services operated by Microsoft itself but does not permit use in the Web IQ and Foundry grounding APIs offered to third parties. This makes it possible to opt out of training while still being used as a source for answers in AI services operated by Microsoft.
For inclusion in regular Bing search, the conventional crawling and indexing conditions continue to apply. Appearing and ranking in search results is a different process from AI answer grounding, and restricting use in AI answers does not directly lead to exclusion from regular search results.
OpenAI, meanwhile, has stated its intent to support the common signal but has not clarified the detailed order of precedence relative to existing crawler directives, or how it would affect use in AI answers. Google's or Microsoft's specifications cannot be assumed to apply to OpenAI, so details released at implementation will need to be checked.
Bing's legacy tags require migration from March 1, 2027
In addition to supporting the common signal, Microsoft Bing is changing its existing settings for restricting AI use. Sites that have used NoArchive or NoCache to restrict AI training or answer generation will need to move to the new directives.
Microsoft announced that from March 1, 2027, it will no longer interpret NoArchive and NoCache as expressions of intent regarding AI training and answer use. This does not mean that every use of these legacy tags is being discontinued; what changes is how they are treated for controlling AI use of content.
Sites that want to keep restricting use in AI answers must publish the new page-level setting by March 1 and, where necessary, register a temporary transition-period setting in Bing Webmaster Tools (BWT). Microsoft plans to offer this temporary setting to sites that are using the legacy tags as of December 15, 2026. However, a similar transition measure will be provided only for answer-use settings; no temporary setting will be offered for AI training.
The BWT temporary setting applies until Bing recrawls the target page and discovers the new directive, or until September 30, 2027, whichever comes first. If a site does not register the temporary setting, answer use reverts to Allow from March 1, 2027 until a new directive is discovered. Once Bing confirms a new directive, it will no longer treat remaining legacy tags as AI-use restrictions.
Note that the date a site changes its settings does not necessarily match the date the change takes effect in Bing. For a new directive to become effective, Bing must crawl the page and detect the change. Even if updates are notified through IndexNow, the timing of the recrawl is not guaranteed.
Sites that use the legacy tags therefore need to prepare with both the February 1, 2027 start of support for the new directive and the March 1 switchover of the old directives in mind. Beyond publishing the new page settings, it is important to confirm whether the site qualifies for the BWT temporary setting so that intended usage restrictions remain in place during the transition.
An AI training opt-out alone does not control use in answers
If the robots.txt common signal becomes widespread, site operators will be able to communicate their AI training policy to multiple companies at once. Less effort spent managing different crawler names and configuration methods for each company is an advantage, particularly for media outlets and large sites with many pages.
However, what is being standardized is mainly the policy on AI training use, not the conditions for use in AI answer generation. Microsoft lets site owners control training and grounding separately, while for Google the combination with Google-Extended matters. For OpenAI, detailed specifications are not yet clear, and consistency with existing settings will need to be judged as each company's implementation becomes known.
At Microsoft, when no applicable directive exists, both AI training and answer-generation use are treated as Allow. Allowing use, however, does not guarantee inclusion in AI answers, traffic to the site, or payment of compensation. These directives themselves also do not establish content licensing agreements or payment terms.
Going forward, what matters for site operators is how many companies end up supporting the common signal, and how training and answer-generation use can each be controlled. The order of precedence relative to existing individual directives, and the time it takes for setting changes to be recognized by crawlers, will also be operational challenges. Sites that use Bing's legacy tags in particular need to migrate ahead of the March 2027 switchover. To put the common expression of intent to practical use in content management, it is essential to keep checking each company's specifications and how settings are being reflected.