The Google AI training control in robots.txt is a shared policy token that lets a site state one AI training preference for several providers at once. Google Search Central, October 2026 announced plans to support it, building on Google-Extended. It is about training permission, not about whether you are cited in answers.
Key takeaways
- Nothing changes in your robots.txt today: Google says support arrives in 2027, so this week is for deciding policy and setting a baseline.
- Training permission and citation eligibility are separate switches. Decide each one on purpose, per provider.
- Do not block Googlebot, OAI-SearchBot or other search crawlers to express a training preference.
- The opt-out is a request under robots.txt, not access control, and it does not remove content already collected.
- Record how often AI engines name and cite you now, so any shift after you change robots.txt can be attributed.
What is Google's new AI training control in robots.txt?
On 30 October 2026, Google said it plans to support a shared AI training control in robots.txt. Per Search Engine Roundtable, October 2026, site owners will use a shared "policy token" to set broad preferences across participating AI providers instead of listing each provider. The work comes from the AIPREF working group at the Internet Engineering Task Force (IETF).
AI Weekly, October 2026 reports that OpenAI, Google, Microsoft and Anthropic are backing the single opt-out token, and Unite.AI, 30 October 2026 reports that OpenAI will honour the signal. Google's own post is the source for Google's plan and its 2027 timing; check each provider's own documentation before you rely on its commitment.
The token and how it is written
According to AI Weekly, October 2026, the token is __usage_ai-training. A site writes a User-Agent line with that token, then Disallow: /, and can add Allow: lines for exceptions.
Treat the syntax as draft. Unite.AI, 30 October 2026 says the Internet-Draft is dated 30 October 2026 and expires on 3 May 2027, so details can still change before you deploy anything.
How it maps to Google-Extended
Google says that if a site opts out with the token and does not name Google-Extended, Google plans to read that as a Google-Extended disallow (Search Engine Roundtable, October 2026). That is the only case the source states. The other rows below are our reading of how robots.txt group rules usually work, not a statement from Google.
| Your robots.txt | Google's planned reading | Basis |
|---|---|---|
| Token disallowed, Google-Extended not mentioned | Treated as Google-Extended disallowed | Stated by Google, as reported |
| Token disallowed, Google-Extended also named with its own rule | Likely the named Google-Extended rule applies | Writer's reading, unconfirmed |
| Only Google-Extended disallowed | Likely unchanged from today | Writer's reading, unconfirmed |
Will blocking AI training hurt my visibility in AI answers?
Not directly for Google Search and OpenAI's ChatGPT search, because training and answering use different controls. For Google there is a wrinkle worth checking.
Google-Extended is a token, not a crawler. Promptwatch, as checked on 31 October 2026, summarises Google's position: a Google-Extended rule does not affect inclusion in Search and is not a ranking signal. AI Overviews and AI Mode are part of Search and rely on Googlebot.
The wrinkle: a glossary entry at Anglera, as checked on 31 October 2026 describes Google-Extended as covering Gemini training and grounding in Gemini Apps and Vertex AI. If that holds, a broad opt-out could reduce how Gemini apps draw on your pages, even while Search is unaffected. Some early coverage claims the opposite. Neither claim is settled until Google ships the control, so test it rather than assume.
Training versus citation, by provider
| Provider | Training control | Answer or search control |
|---|---|---|
| Google-Extended (and the token from 2027) | Googlebot, plus page-level snippet controls | |
| OpenAI | GPTBot | OAI-SearchBot for ChatGPT search; ChatGPT-User for user-initiated fetches |
Momentic, checked against operator docs on 19 October 2026, notes that OpenAI treats each robots.txt setting as independent, so a site can allow OAI-SearchBot and disallow GPTBot. For ChatGPT-User, OpenAI says robots.txt rules may not apply, because a user initiated the action.
In plain terms: Google-Extended is the control that Google describes as governing Gemini training and certain grounding. Googlebot is the crawler that feeds Google Search, including AI Overviews. Blocking the first is a training decision; blocking the second removes you from Search.
What the control does not do
- It is not enforcement. OpenAI frames it as a coordination mechanism, not a security control, so non-compliant scrapers can ignore it (AI Weekly, October 2026).
- It is not retroactive. Updating robots.txt does not remove material already collected into datasets or trained models (same source).
- It is not live. Google's support is planned for 2027.
Google also notes the Search Console opt-out for Search generative AI exists separately (Search Engine Roundtable, October 2026).
What to check this week
- Read your live robots.txt. Note every AI-related group: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. The free AI Crawler Checker shows which of these can reach your site and generates a corrected file.
- Check the CDN and WAF. A bot-protection rule can block search crawlers even when robots.txt allows them. The AI Crawler Policy Kit covers this.
- Write the policy in one sentence per provider. For example: allow search and user-initiated fetching, decide training separately.
- Set the baseline. Before changing anything, record mention rate and citations on a fixed prompt set (see the next section).
- Do not rush the token. The draft expires in May 2027 and may change. Keep existing named rules in place.
- Review your llms.txt. It does not set training permission, but it is part of the same crawler-facing set of files. The free llms.txt Generator and Validator builds one from your sitemap and checks existing links. Our llms.txt guide explains what it does and does not do.
How to measure whether visibility shifts afterwards
A change in robots.txt is an intervention, so measure it like one. Run the same buyer prompts on each engine, several times, before and after. AI answers vary from run to run, so a single before and after check proves little; see AI citation stability for how many runs to sample.
| Step | What to record | Why |
|---|---|---|
| 1. Baseline, 2 weeks | Mention rate, citation rate and source URLs per engine | Sets the normal range of noise |
| 2. Change one thing | Date, file diff, which token or crawler | Keeps cause traceable |
| 3. Re-measure, 2 to 4 weeks | Same prompts, same engines | Allows crawl and index lag |
| 4. Judge | Change versus baseline range | Call it a shift only if it exceeds noise |
Separate mention from citation: an engine can name you without linking your page, and the reverse. The citation versus brand mention post explains why each needs different work. Check Gemini on its own, given the grounding question above; the Gemini visibility tracker guide shows how.
Where Noma fits
Noma by Nurdd asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, your buyers' questions every day. It reports whether your brand is named, how high, and which pages the answers cite, per engine. Because the prompt set stays fixed, you can compare the weeks before and after a robots.txt change and see whether any engine moved. Noma also reports which AI crawlers fetch your pages.
See it on your own brand: book a demo or start the 3-day free trial. A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged.
Mistakes to avoid
- Blocking
*to stop training. It also blocks search and user fetchers, which can remove you from answers. - Changing several rules at once. You will not know which one moved the numbers.
- Reading one run as a result. Sample repeatedly and compare to the baseline range.
- Treating the opt-out as a deletion request. It governs future use only.
How we researched this
We reviewed Google's Search Central announcement, trade coverage from Search Engine Roundtable, reporting on the provider commitments from AI Weekly and Unite.AI, and third-party crawler references summarising operator documentation. We did not test the token ourselves, since support has not shipped. Facts, dates and names were checked on 2026-10-31. Where sources disagree, notably on Gemini grounding, we say so, and we label our own readings of the rules.
What to do next
Audit your robots.txt now, write your training-versus-citation policy, and capture a baseline before 2027. To track the effect across 13 AI engines, book a demo or start the free trial. For background, read how AI engines choose citations.
See what AI engines say about your brand
Free · no account · or start a 3-day trial
Questions people ask
What is Google's new AI training control in robots.txt?
It is a shared policy token, reported as __usage_ai-training, that lets a site state one AI training preference for several providers in robots.txt. Google announced on 30 October 2026 that it plans to support it in 2027. Google says it will read an opt-out as a Google-Extended disallow unless you name Google-Extended yourself.
Will blocking AI training hurt my visibility in AI answers?
For Google Search and ChatGPT search, training controls are separate from the search crawlers, so blocking training should not remove you from them. One open point is Gemini: Google-Extended is described by a third party as covering some grounding in Gemini Apps too. Measure mentions and citations before and after to confirm.
Do I need to change robots.txt now?
No. Google's support is planned for 2027 and the draft expires on 3 May 2027, so details may change. Use this period to audit your current rules, set a policy and record a baseline.
Does the new control remove my content from existing AI models?
No. Updating robots.txt does not remove material already collected into datasets or trained models. It also relies on providers choosing to comply, so it is not access control.
How do I allow ChatGPT citations but opt out of training?
OpenAI treats its robots.txt settings independently. You can disallow GPTBot for training and allow OAI-SearchBot so pages can surface in ChatGPT search. ChatGPT-User fetches are user-initiated, so robots.txt rules may not apply to them.
How can I tell whether a robots.txt change affected my AI visibility?
Run a fixed set of buyer prompts on each engine several times before the change, then repeat after allowing time for crawling. Call it a shift only if the change exceeds the normal run-to-run range.