OpenAI and Google announced on October 30 that they will support a shared `robots.txt` signal allowing website operators to communicate, through one rule, that they do not authorize their content to be used for generative-model training. Microsoft and Anthropic also back the initiative, according to *The Verge*. The change is intended to replace the burden of configuring separate instructions for every provider with one preference that multiple companies can interpret consistently.[1][2][4]

The proposal uses the policy token `__usage_ai-training`. In practical terms, a website could add that identifier to its `robots.txt` file and block access across the domain or permit selected paths. The technical draft was submitted by Anthropic’s Kevin Kelley in work connected to the Internet Engineering Task Force’s AI Preferences (AIPREF) group.[3]

The initiative’s status requires a clear caveat: it is not yet an IETF-approved standard. The official record classifies it as an individual Internet-Draft with no formal IETF endorsement and no assigned RFC status. That distinction matters because the announcement is a voluntary commitment by specific providers, not a universal rule or a legal obligation covering every AI developer.[3]

Implementation will not happen on one common date. OpenAI says it aims to honor the signal “in the coming months.” Google plans to add support in 2027. *The Verge* reports that Anthropic has already adopted it and that Microsoft intends to support it next year. Existing provider-specific controls remain in force while those changes move into production.[1][2][4]

Google explained that when it encounters the shared opt-out token and a website has not issued a specific instruction for Google-Extended, it plans to interpret the token as though Google-Extended had been disallowed. Website operators will still be able to combine the broad signal with individual rules that allow or deny particular providers. Google also noted that sites allowing Google-Extended can be eligible to appear as linked sources in grounded Gemini responses.[2]

OpenAI likewise said the shared signal will not replace its current crawler controls. The company says it will continue to honor its existing provider-specific instructions and describes the common option as an additional layer that reduces administrative work for publishers and website operators.[1]

The signal is designed to express a preference about training foundational generative models. It does not withdraw data already collected, compel providers that have not agreed to participate, or guarantee that an unknown crawler will obey the instruction. It should not automatically be treated as a control for search indexing, information retrieval, or grounding in generated answers, either; those uses may be governed by different rules depending on the provider and the website’s configuration.[1][2][3]

The development is consequential because it aligns four of the most influential AI companies around a common syntax, but its practical value will depend on verifiable implementation and broader adoption. For publishers, the immediate benefit is operational as much as conceptual: one uniform preference reduces fragmented configuration. The central limitation remains unchanged, however: `robots.txt` communicates a policy; it is not a technical barrier against an actor that chooses to ignore it.

The cautious reading is that the industry has taken a step toward interoperability, not that it has settled disputes over consent, licensing, or compensation for training data. The draft creates a simpler way to declare a choice. The next tests will be when each provider activates support, how each documents compliance, and whether other developers join the effort.

Sources

[1] Read More — Making it easier for websites to express AI training preferences — OpenAI
[2] Read More — Supporting a shared AI training control for websites — Google
[3] Read More — Machine-Readable Training Preference Signal — IETF Datatracker
[4] Read More — The biggest names in AI are backing a new way to opt out of scraping — The Verge