What Is __usage_ai-training? The New robots.txt AI Signal

__usage_ai-training is a draft robots.txt token that lets you tell participating AI providers, in one group of rules, whether your content may be used to train models. On October 30, 2026, Google said it aims to support it in 2027, and Microsoft said Bing will honor it from February 1, 2027.

Until now, opting out of AI training meant listing each company's crawler or product token separately. This article explains what was announced, what the new line looks like, what it does not do, and the practical steps worth taking now while the standard is still a draft.

What happened

Several announcements landed on the same day:

Search Engine Roundtable's coverage also linked to an OpenAI post on the subject. We could not open that page, so we do not describe OpenAI's plans here.

What Google said

Google's post describes "a shared 'policy token' within robots.txt to set broad preferences across participating AI providers." The key behavior for site owners: if you use the shared token to opt out of training "(and Google-Extended is not explicitly specified)", Google plans to treat that as if you had disallowed Google-Extended, its existing AI training control.

Google also said: "Existing company-specific controls will continue to function alongside this new joint signal." So a Google-Extended rule you already have still works, and a specific rule for a named provider can override the broad preference for that provider.

Google was careful to call this a draft. Its post says other AI providers had not yet adopted it at the time and that the wider AIPREF vocabulary, beyond the training definition, is still being discussed. Google ties the proposal to the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary way for publishers to state machine-readable preferences.

What the new line looks like

The IETF draft adds no new directives. You use the policy token as if it were a user agent name, then write ordinary Allow and Disallow rules. The draft's own example is:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against AI training for the whole site, except the /press-kit/ folder.

Three details matter:

1. Spelling is exact. Two leading underscores, then usage, another underscore, then ai-training.
2. The wildcard group does not count. The draft says User-Agent: * groups "MUST NOT be interpreted as matching the policy token." A blanket rule for all bots does not set a training preference, and a training rule does not block crawling.
3. It is not a crawler. No bot sends __usage_ai-training in its requests. The draft says the token "need not appear in the User-Agent header field."

The meaning of "AI training" comes from the IETF AIPREF working group's vocabulary: using an asset to modify the learned parameters of a generative AI model.

What it does not do

Read the token as a training preference only. Based on the Google and Microsoft posts:

• It does not remove you from Google Search or Bing results. Google's crawler documentation says "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search," and Microsoft says traditional Bing indexing and ranking are not affected by its new preferences.
• It does not control AI answers in search. Microsoft handles "grounding" (using your page to help write an AI answer) with separate page-level controls. Google's post covers training, and says sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses through grounding.
• It is not a paywall or license. Microsoft's post says preferences do not create licensing or payment arrangements.
• It is not finished. The Datatracker page notes the draft is an individual submission, "not endorsed by the IETF," with "no formal standing." Details could change before 2027.

Who this affects

• Publishers, bloggers and content-led businesses that have strong views on AI training, either for or against.
• Agencies and developers who maintain robots.txt files for clients and currently keep long lists of AI crawler names.
• Sites that already block Google-Extended or GPTBot. Nothing breaks, but you may want to simplify later.
• Sites that want AI visibility. If you value appearing in AI tools, be careful not to copy a blanket training block without understanding what each provider ties to it.

What to do now

Nothing here is urgent. Google's support is targeted for 2027 and Microsoft's starts on February 1, 2027. Use the time to decide your policy rather than to rush a change.

1. Write down your preference. Decide, in plain language, whether you are happy for your content to train AI models, and whether some folders (press kit, docs, product pages) should be treated differently from others.
2. Audit what you block today. Run your domain through our AI crawler checker to see which AI user agents your robots.txt allows or blocks, and keep a copy of the current file.
3. Keep company-specific rules for now. Google and Anthropic both describe the shared signal as working alongside existing controls, not replacing them. Removing a Google-Extended or other named rule today would leave a gap until providers actually support the new token.
4. Add the token only when you mean it. If your decision is to opt out of training, adding a __usage_ai-training group now does no harm and will be read as providers adopt it. If you want to stay in training data, you do not need to add anything; Microsoft's post says both of its preferences default to Allow when nothing is published.
5. Test the file after every edit. A typo in a robots.txt file can block far more than you intended. Check specific URLs against your rules with our robots.txt tester, and see our guide to testing robots.txt rules.
6. Check your CDN or host settings. Some hosts and CDNs manage robots.txt or AI bot rules for you. Our article on Cloudflare's AI crawler controls covers one common setup. Make sure the file your visitors' bots actually fetch is the one you edited.
7. Revisit in early 2027. Watch the Google Search Central blog and the IETF draft for changes to the token name or behavior before relying on it.

If you would rather generate a clean starting file, our robots.txt generator can help, and the free tools at /tools need a free account. Our broader guide on whether to block or allow AI crawlers covers the trade-offs.

Example: opt out of training, keep everything else

Here is one way a small business might combine rules. It keeps normal crawling open, states a training preference for participating providers, and keeps an existing Google-Extended rule:

User-agent: *
Allow: /

User-Agent: __usage_ai-training
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

This is an illustration, not a recommendation for every site. If you want Gemini Apps grounding, Google's post says sites must specifically allow Google-Extended, so a site with that goal would make a different choice.

Frequently asked questions

What is __usage_ai-training in robots.txt?

It is a "policy token" proposed in an IETF draft published on October 30, 2026. You put it in a User-Agent line and add normal Allow or Disallow rules to state whether your content may be used for AI training. It is a preference label, not the name of a real crawler.

Does Google support the AI training token yet?

Not yet. Google said on October 30, 2026 that it aims to support the control in 2027, and its crawler documentation now says "early 2027." Until then, Google-Extended remains Google's AI training control.

Will blocking AI training hurt my Google rankings?

Google-Extended, which the new token maps to, does not affect Google Search ranking. Microsoft also says its training and grounding preferences do not change eligibility for traditional Bing results. Keep normal search crawlers allowed in your User-agent: * group.

Does User-agent: * cover the AI training token?

No. The draft states that wildcard groups must not be interpreted as matching the policy token. To express a training preference you need a group that names __usage_ai-training explicitly.

Should I remove my GPTBot or Google-Extended rules?

Not now. Google says company-specific controls continue to work alongside the shared signal, and providers have not finished implementing it. Keep named rules until each provider confirms support, then simplify if you want.

Is the shared AI training signal an official standard?

No. It is an individual Internet-Draft that the IETF Datatracker says is not endorsed by the IETF and has no formal standing. Google, Microsoft and Anthropic have said they will support it, but the details may change as the standards work continues.

Sources

• Google Search Central: Supporting a shared AI training control for websites (October 30, 2026)
• IETF Datatracker: Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00
• Google crawling documentation changelog
• Google: common crawlers, Google-Extended
• Microsoft Bing Blog: Your content, your choice: clearer preferences for training and grounding (October 30, 2026)
• IETF ai-control mailing list: Kevin Kelley message on Anthropic implementation (October 30, 2026)
• Search Engine Roundtable: Google to adopt new AI training control