Google and OpenAI Back a Shared robots.txt AI-Training Token From an Anthropic Draft, and Neither Has Shipped It
A.I. / news
Google and OpenAI Back a Shared robots.txt AI-Training Token From an Anthropic Draft, and Neither Has Shipped It
The token __usage_ai-training is an individual IETF draft dated October 30. Google aims for 2027, and OpenAI says in the coming months.
Google and OpenAI have each said they will honour a shared robots.txt token that lets a site opt out of AI training in one line, and neither has shipped it as of Wednesday. The token, __usage_ai-training, comes from an individual Internet-Draft dated October 30, 2026.
Google's side is on the record. John Mueller, Google Search Advocate, wrote in a post on Google's Search Central blog that Google aims to support the shared control with other AI providers in 2027. OpenAI's commitment reaches us through ai-tldr.dev, a summary site, which reports that OpenAI said on October 30 it will implement the signal "in the coming months." OpenAI's own page was not fetchable on Wednesday.
What the draft says
The draft is by Kevin Kelley of Anthropic. It is titled "Machine-Readable Training Preference Signal" and expires May 3, 2027. It is an individual submission that the IETF has not endorsed, and it carries no formal standing in the standards process.
It adds no new directive. A site writes an ordinary robots.txt group and names the token as the user agent:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That expresses a preference against AI training everywhere except under /press-kit/. The rules are narrow:
- The token is matched exactly, ignoring case.
- A
User-Agent: *group must not be read as matching it. - A file with no group for the token expresses no preference.
- The token is not a crawl directive, and private agreements can override it.
The draft also warns that parsers which do not implement it will ignore the group, and that some match tokens more loosely than specified.
Where Google and OpenAI stand
Google's post says a site that opts out with the shared token and does not name Google-Extended will be treated as having disallowed Google-Extended. Existing company controls, including Google-Extended, keep working beside the shared signal. The post does not mention AI Overviews. It says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini apps.
OpenAI's crawler documentation did not mention the token on Wednesday. It lists three agents, each controlled separately. GPTBot crawls content that may train its foundation models. OAI-SearchBot decides whether a site appears in ChatGPT search. ChatGPT-User handles actions a user starts, and OpenAI says robots.txt rules may not apply to it.
| Agent or signal | What it controls | Status on Nov 4 |
|---|---|---|
| GPTBot | Training use of crawled content | Live |
| OAI-SearchBot | Appearing in ChatGPT search | Live |
| Google-Extended | Google's AI training use | Live |
__usage_ai-training | Shared training opt-out | Draft, no live support |
Three drafts, one working group
The IETF's AI Preferences working group lists two working-group documents, both aimed at Proposed Standard. The Kelley draft sits outside them as an individual submission.
| Draft | Date | Status |
|---|---|---|
| draft-ietf-aipref-attach-05 | 2026-08-18 | Working-group document |
| draft-ietf-aipref-vocab-08 | 2026-09-13 | Working-group document |
| draft-kelley-aipref-training-preference-signal-00 | 2026-10-30 | Individual draft |
Until a crawler reads the token, a site that wants a block today still needs a separate group for each company. Model makers covered here, such as Baidu with Unlimited-OCR-2 and Nvidia with Nemotron 3.5 Transcribe, publish models whose training data is a separate question from any robots.txt file.
Google says it will post updates on progress. The draft lapses on May 3, 2027 unless it is revised or replaced.
Sources
More in A.I.
- 01John Langford Says He Left AI Frontiers After Microsoft AI ReorgMicrosoft's lab page confirms the Sept. 14 move into Microsoft AI, while Langford's blog post says the new owner's philosophy is scaling and not research.
- 02Nemotron 3.5 Transcribe Posts 4.62 Word Error Rate Across Seven Languages, NVIDIA's Own NumberThe offline speech model has 4.3 billion active parameters, 40 languages and a licence that permits commercial use.
- 03Cloudflare's Security Audit Skill Has an Open Issue Saying a Made-Up Finding Can Pass ValidationThe MIT-licensed skill runs six phases of agent audits. Its tracker shows 43 open issues, including a profile-wipe report and a Windows failure.
- 04Anthropic's Agent Memory Study Puts Opus 5.5 at 0.838 With Memory and 0.655 WithoutThe Nov. 6 post tests notes, wikis and offline dreaming, and reports where a wiki beat search and where it did not.