# Google AI Training Control in robots.txt: What to Do

> Google will support a shared AI training control in robots.txt in 2027. See what it does, what it leaves alone, and a checklist to protect AI citations.

- Author: Simran Rawat, Growth Executive, Nurdd
- Published: 2026-10-31
- Publisher: Noma by Nurdd (https://nomahq.app)
- Original page: https://nomahq.app/blog/google-ai-training-control-robots-txt

The Google AI training control in robots.txt is a shared policy token that lets a site state one AI training preference for several providers at once. [Google Search Central, October 2026](https://developers.google.com/search/blog/2026/10/ai-training-control) announced plans to support it, building on Google-Extended. It is about training permission, not about whether you are cited in answers.

## Key takeaways

- Nothing changes in your robots.txt today: Google says support arrives in 2027, so this week is for deciding policy and setting a baseline.
- Training permission and citation eligibility are separate switches. Decide each one on purpose, per provider.
- Do not block Googlebot, OAI-SearchBot or other search crawlers to express a training preference.
- The opt-out is a request under robots.txt, not access control, and it does not remove content already collected.
- Record how often AI engines name and cite you now, so any shift after you change robots.txt can be attributed.

## What is Google's new AI training control in robots.txt?

On 30 October 2026, Google said it plans to support a shared AI training control in robots.txt. Per [Search Engine Roundtable, October 2026](https://www.seroundtable.com/google-ai-training-control-42430.html), site owners will use a shared "policy token" to set broad preferences across participating AI providers instead of listing each provider. The work comes from the AIPREF working group at the Internet Engineering Task Force (IETF).

[AI Weekly, October 2026](https://aiweekly.co/alerts/openai-google-microsoft-anthropic-back-one-ai-opt-out-token) reports that OpenAI, Google, Microsoft and Anthropic are backing the single opt-out token, and [Unite.AI, 30 October 2026](https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/) reports that OpenAI will honour the signal. Google's own post is the source for Google's plan and its 2027 timing; check each provider's own documentation before you rely on its commitment.

### The token and how it is written

According to [AI Weekly, October 2026](https://aiweekly.co/alerts/openai-google-microsoft-anthropic-back-one-ai-opt-out-token), the token is `__usage_ai-training`. A site writes a `User-Agent` line with that token, then `Disallow: /`, and can add `Allow:` lines for exceptions.

Treat the syntax as draft. [Unite.AI, 30 October 2026](https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/) says the Internet-Draft is dated 30 October 2026 and expires on 3 May 2027, so details can still change before you deploy anything.

### How it maps to Google-Extended

Google says that if a site opts out with the token and does not name Google-Extended, Google plans to read that as a Google-Extended disallow ([Search Engine Roundtable, October 2026](https://www.seroundtable.com/google-ai-training-control-42430.html)). That is the only case the source states. The other rows below are our reading of how robots.txt group rules usually work, not a statement from Google.

| Your robots.txt | Google's planned reading | Basis |
| --- | --- | --- |
| Token disallowed, Google-Extended not mentioned | Treated as Google-Extended disallowed | Stated by Google, as reported |
| Token disallowed, Google-Extended also named with its own rule | Likely the named Google-Extended rule applies | Writer's reading, unconfirmed |
| Only Google-Extended disallowed | Likely unchanged from today | Writer's reading, unconfirmed |

Google's announcement of the shared AI training control, dated 30 October 2026. Source: Google Search Central.

## Will blocking AI training hurt my visibility in AI answers?

Not directly for Google Search and OpenAI's ChatGPT search, because training and answering use different controls. For Google there is a wrinkle worth checking.

Google-Extended is a token, not a crawler. [Promptwatch, as checked on 31 October 2026](https://promptwatch.com/bots/google-extended), summarises Google's position: a Google-Extended rule does not affect inclusion in Search and is not a ranking signal. AI Overviews and AI Mode are part of Search and rely on Googlebot.

The wrinkle: a glossary entry at [Anglera, as checked on 31 October 2026](https://www.anglera.com/glossary/google-extended) describes Google-Extended as covering Gemini training *and* grounding in Gemini Apps and Vertex AI. If that holds, a broad opt-out could reduce how Gemini apps draw on your pages, even while Search is unaffected. Some early coverage claims the opposite. Neither claim is settled until Google ships the control, so test it rather than assume.

### Training versus citation, by provider

| Provider | Training control | Answer or search control |
| --- | --- | --- |
| Google | Google-Extended (and the token from 2027) | Googlebot, plus page-level snippet controls |
| OpenAI | GPTBot | OAI-SearchBot for ChatGPT search; ChatGPT-User for user-initiated fetches |

[Momentic, checked against operator docs on 19 October 2026](https://momenticmarketing.com/blog/ai-search-crawlers-bots), notes that OpenAI treats each robots.txt setting as independent, so a site can allow OAI-SearchBot and disallow GPTBot. For ChatGPT-User, OpenAI says robots.txt rules may not apply, because a user initiated the action.

In plain terms: Google-Extended is the control that Google describes as governing Gemini training and certain grounding. Googlebot is the crawler that feeds Google Search, including AI Overviews. Blocking the first is a training decision; blocking the second removes you from Search.

## What the control does not do

- **It is not enforcement.** OpenAI frames it as a coordination mechanism, not a security control, so non-compliant scrapers can ignore it ([AI Weekly, October 2026](https://aiweekly.co/alerts/openai-google-microsoft-anthropic-back-one-ai-opt-out-token)).
- **It is not retroactive.** Updating robots.txt does not remove material already collected into datasets or trained models (same source).
- **It is not live.** Google's support is planned for 2027.

Google also notes the Search Console opt-out for Search generative AI exists separately ([Search Engine Roundtable, October 2026](https://www.seroundtable.com/google-ai-training-control-42430.html)).

## What to check this week

1. **Read your live robots.txt.** Note every AI-related group: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. The free [AI Crawler Checker](https://nomahq.app/tools/ai-crawler-checker) shows which of these can reach your site and generates a corrected file.
2. **Check the CDN and WAF.** A bot-protection rule can block search crawlers even when robots.txt allows them. The [AI Crawler Policy Kit](https://nomahq.app/resources/ai-crawler-policy-kit) covers this.
3. **Write the policy in one sentence per provider.** For example: allow search and user-initiated fetching, decide training separately.
4. **Set the baseline.** Before changing anything, record mention rate and citations on a fixed prompt set (see the next section).
5. **Do not rush the token.** The draft expires in May 2027 and may change. Keep existing named rules in place.
6. **Review your llms.txt.** It does not set training permission, but it is part of the same crawler-facing set of files. The free [llms.txt Generator and Validator](https://nomahq.app/tools/llms-txt) builds one from your sitemap and checks existing links. Our [llms.txt guide](https://nomahq.app/blog/llms-txt-guide) explains what it does and does not do.

## How to measure whether visibility shifts afterwards

A change in robots.txt is an intervention, so measure it like one. Run the same buyer prompts on each engine, several times, before and after. AI answers vary from run to run, so a single before and after check proves little; see [AI citation stability](https://nomahq.app/blog/ai-citation-stability) for how many runs to sample.

| Step | What to record | Why |
| --- | --- | --- |
| 1. Baseline, 2 weeks | Mention rate, citation rate and source URLs per engine | Sets the normal range of noise |
| 2. Change one thing | Date, file diff, which token or crawler | Keeps cause traceable |
| 3. Re-measure, 2 to 4 weeks | Same prompts, same engines | Allows crawl and index lag |
| 4. Judge | Change versus baseline range | Call it a shift only if it exceeds noise |

Separate mention from citation: an engine can name you without linking your page, and the reverse. The [citation versus brand mention](https://nomahq.app/blog/citation-vs-brand-mention) post explains why each needs different work. Check Gemini on its own, given the grounding question above; the [Gemini visibility tracker guide](https://nomahq.app/blog/gemini-visibility-tracker) shows how.

## Where Noma fits

Noma by Nurdd asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, your buyers' questions every day. It reports whether your brand is named, how high, and which pages the answers cite, per engine. Because the prompt set stays fixed, you can compare the weeks before and after a robots.txt change and see whether any engine moved. Noma also reports which AI crawlers fetch your pages.

See it on your own brand: [book a demo](https://nomahq.app/demo) or [start the 3-day free trial](https://platform.nomahq.app/signup). A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged.

## Mistakes to avoid

- **Blocking `*` to stop training.** It also blocks search and user fetchers, which can remove you from answers.
- **Changing several rules at once.** You will not know which one moved the numbers.
- **Reading one run as a result.** Sample repeatedly and compare to the baseline range.
- **Treating the opt-out as a deletion request.** It governs future use only.

## How we researched this

We reviewed Google's Search Central announcement, trade coverage from Search Engine Roundtable, reporting on the provider commitments from AI Weekly and Unite.AI, and third-party crawler references summarising operator documentation. We did not test the token ourselves, since support has not shipped. Facts, dates and names were checked on 2026-10-31. Where sources disagree, notably on Gemini grounding, we say so, and we label our own readings of the rules.

## What to do next

Audit your robots.txt now, write your training-versus-citation policy, and capture a baseline before 2027. To track the effect across 13 AI engines, [book a demo](https://nomahq.app/demo) or [start the free trial](https://platform.nomahq.app/signup). For background, read [how AI engines choose citations](https://nomahq.app/blog/how-ai-engines-choose-citations).