An AI bot policy in robots.txt sorts every AI crawler into one of three jobs: training, retrieval (search indexing) or user-triggered fetching. Each job has a different cost and a different upside. You can block GPTBot, which trains models, and still allow OAI-SearchBot, which feeds ChatGPT search, so your pages can still be cited.
Key takeaways
- Decide by job, not by company. Training bots, retrieval bots and user-triggered fetchers each need their own line in robots.txt.
- Blocking a training bot (GPTBot, ClaudeBot, Google-Extended, CCBot) is a licensing choice. Blocking a retrieval bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot) is a visibility choice.
- User-triggered fetchers such as ChatGPT-User and Perplexity-User generally ignore robots.txt, so a Disallow line is a weak control for them.
- Your CDN or firewall can block bots that robots.txt allows. Check both layers before you conclude anything.
- Measure the effect: record which engines name and cite you before the change, then compare after a few weeks.
What is the difference between AI training and retrieval crawlers?
A training crawler collects public pages that may be used to build or improve a model. A retrieval crawler indexes pages so an AI search product can find and cite them when someone asks a question. A user-triggered fetcher loads one page because a person asked the assistant to read it.
Search Engine Journal's Chris Green puts the stakes plainly in Search Engine Journal, November 2026: "we need to move from block lists to deliberate bot policies." His point is that blocking a training crawler has different consequences from blocking one that retrieves content for search answers.
| Job | What the bot does | Examples | What blocking costs you |
|---|---|---|---|
| Training | Collects content that may be used to train future models | GPTBot, ClaudeBot, Google-Extended, CCBot | Little direct citation loss. You opt out of model training. |
| Retrieval / search indexing | Builds the index an AI search product answers from | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You can drop out of that engine's answers and citations. |
| User-triggered fetch | Loads a page when a person asks the assistant to read it | ChatGPT-User, Claude-User, Perplexity-User | Users cannot get the assistant to summarise or compare your page. Often not enforceable in robots.txt anyway. |
The definitions above follow how OpenAI's bots are described in Natie Limelech, October 2026: GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for fetches after a user action. For the wider picture of how engines choose what to cite, see how AI engines choose citations.
Should I block GPTBot but allow OAI-SearchBot?
Yes, if you want to opt out of OpenAI model training and still appear in ChatGPT search. The two bots are controlled by separate robots.txt tokens. According to GeoToolbox, November 2026, summarising OpenAI's documentation, sites opted out of OAI-SearchBot will not appear in ChatGPT search answers, though they can still show up as navigational links. Blocking GPTBot leaves that visibility intact.
Two cautions. First, a GPTBot request in your logs does not prove your content was trained on or will be cited, so do not read crawl counts as visibility. Second, the opposite mistake is common: a blanket rule that blocks every "GPT" or "AI" user agent will catch OAI-SearchBot too. Write explicit groups for each token.
The same split exists at other vendors. SEOmator, November 2026 describes Anthropic's three bots: ClaudeBot (training), Claude-SearchBot (search indexing) and Claude-User (user-requested fetches), each controllable on its own. Blocking Claude-SearchBot may reduce your visibility and accuracy in Claude's search results.
Which bots should I allow or block? A per-bot policy table
This table assumes the goal most brands have: stay citable in AI answers, and decide separately whether to license content for training. Treat the default column as a starting point, not a rule. Bot names and behaviour change, so confirm each against the vendor's own documentation before deploying.
| Bot (robots.txt token) | Operator | Job | Default for a brand that wants citations | Notes |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Your call: block to opt out of training | Blocking does not remove you from ChatGPT search |
| OAI-SearchBot | OpenAI | Search indexing | Allow | Needed to appear in ChatGPT search answers |
| ChatGPT-User | OpenAI | User-triggered fetch | Allow | Fires when someone asks ChatGPT to read a page |
| ClaudeBot | Anthropic | Training | Your call: block to opt out | Separate from Claude-SearchBot |
| Claude-SearchBot | Anthropic | Search indexing | Allow | Blocking may reduce visibility in Claude's search results |
| Claude-User | Anthropic | User-triggered fetch | Allow | User-requested fetches |
| PerplexityBot | Perplexity | Search indexing | Allow | Indexes pages for Perplexity's search corpus |
| Perplexity-User | Perplexity | User-triggered fetch | Allow | Perplexity says it generally ignores robots.txt, per GeoToolbox |
| Google-Extended | Training / Gemini grounding token | Your call | A robots.txt token, not a separate crawler, per GeoToolbox. Googlebot stays separate. | |
| CCBot | Common Crawl | Training | Your call: block to opt out | Its dataset feeds many models |
Sources for the table: GeoToolbox, November 2026, SEOmator, November 2026 and Natie Limelech, October 2026, as checked on 2026-11-15. Google's own list of common crawlers, including the Google-Extended token, is on Google Search Central, 2026.
How do I write the robots.txt rules?
Group rules by job. Each group names the tokens it covers and gives one instruction. The pattern below blocks training and keeps retrieval open.
- Training group: User-agent lines for GPTBot, ClaudeBot, CCBot and Google-Extended, followed by Disallow: / . Omit this group if you are happy to be used for training.
- Retrieval group: User-agent lines for OAI-SearchBot, Claude-SearchBot and PerplexityBot, followed by Allow: / . An explicit group makes your intent visible to anyone auditing the file.
- User-triggered group: User-agent lines for ChatGPT-User, Claude-User and Perplexity-User, followed by Allow: / . Treat this as documentation of intent, not enforcement.
- Private paths: keep Disallow lines for checkout, account and internal search pages under every group, so allowing a bot does not open areas you never wanted crawled.
Rules are matched per user-agent group, so a bot follows the most specific group that names it, not a stricter group meant for another bot. Test the file with the free AI Crawler Checker, which shows whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google AI can reach your site and generates a corrected robots.txt.
Why robots.txt alone does not decide who gets in
Robots.txt is a request, not a lock. Two things sit between your file and what actually happens.
User-triggered fetchers often ignore it
Google's documentation on Google Search Central, 2026 says that because a user requested the fetch, user-triggered fetchers ignore robots.txt rules. OpenAI and Perplexity describe their user-triggered bots in similar terms, according to GeoToolbox, which says Perplexity-User generally ignores robots.txt. Plan on that: if a page must stay private, use authentication, not a Disallow line.
Your CDN can contradict your file
Green's Search Engine Journal piece warns that enforcement happens in CDNs, web application firewalls and bot mitigation services, which can contradict robots.txt without the SEO team knowing. A file that allows OAI-SearchBot is no use if the firewall returns 403 to it. The fix is a log check: filter requests by user agent and confirm status codes. Our guide to tracking AI crawlers with logs and analytics walks through it, and this post on ChatGPT agents blocked from a site covers the symptoms.
Compliance is also uneven. A Tollbit report cited by Digiday, relayed in Search Engine Journal, June 2026, found that 30% of total AI bot scrapes did not comply with explicit robots.txt rules. Robots.txt expresses policy. Verification (IP ranges, reverse DNS) enforces it. Google publishes both, as its verification page shows.
What are other sites doing?
Many publishers block more than they intend. Search Engine Journal, January 2026 reported a study of 100 top US and UK news sites' robots.txt files: 79% block at least one AI training bot, and 71% also block at least one retrieval or live search bot. It also reported that 67% blocked PerplexityBot.
News publishers have licensing reasons to block. A software company, retailer or local business usually does not. The 71% figure is the lesson: retrieval bots are blocked by sites that likely meant to block only training. Count this as a check against your own file.
Share of crawl traffic also differs sharply by bot. SEOmator reports Claude-SearchBot at 1.8% of AI-bot requests in October 2026. A small share of requests does not mean a small effect on citations, so do not drop a bot because it looks quiet in your logs.
A decision framework: four questions per bot
Run each bot through these in order. Stop at the first answer that settles it.
- Does it retrieve for an answer engine you want to appear in? If yes, allow it. Your buyers ask these engines for recommendations, and a blocked retrieval bot is a missing page in the index.
- Is it triggered by a user? If yes, allow it unless the content is private, and use authentication for anything private. robots.txt will not reliably stop it.
- Is it a training crawler? This is a business decision. Block if your content is licensed, gated or your main product. Allow if reach matters more and you have no licensing model.
- Can you verify it? If the bot publishes IP ranges, check requests against them. If it does not, expect spoofing and apply rate limits at the CDN.
| Business situation | Training bots | Retrieval bots | User-triggered |
|---|---|---|---|
| B2B SaaS that wants to be recommended | Allow or block (low stakes) | Allow | Allow |
| Publisher selling content licences | Block | Allow if citations drive subscriptions, else block | Allow, accept leakage |
| Ecommerce brand in AI shopping | Allow or block | Allow, including product pages | Allow, so agents can compare products |
| Gated or paid content | Block | Block on gated paths only | Protect with login |
In every row, split by path as well as by bot. Allowing retrieval on /blog/ and /products/ while blocking /account/ is better than a site-wide decision either way. For agent behaviour specifically, read AI agent website access: block, allow or serve them.
How to measure what the policy changed
A robots.txt edit is only useful if you can see its effect on answers. Use a before-and-after test.
- Baseline first. Before changing anything, record for 20 to 50 buyer questions which engines name your brand, which competitors they name instead, and which of your pages they cite. See how to choose prompts to track.
- Change one group at a time. If you block training and unblock retrieval together, you cannot attribute a move to either.
- Confirm access in logs. Look for 200 responses to the retrieval bots within days of the change.
- Re-run the same prompts after a few weeks and compare mentions and citations per engine. Engines differ, so read them separately; why engines disagree explains the gaps.
Expect noise. AI answers vary between runs, which is why you repeat prompts and compare ranges, as covered in AI citation stability.
Where Noma fits
Noma by Nurdd asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, the questions your buyers ask, every day. It shows where your brand is named, where competitors are named instead, and which pages the answers draw on. After a robots.txt change, that gives you a before-and-after view of mentions and cited pages per engine, without sampling by hand.
See it on your own brand: book a demo or start the 3-day free trial. A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged. Prefer a quick look first? The free brand audit checks whether ChatGPT, Perplexity and Gemini name you.
Common mistakes to avoid
- One wildcard for "AI". Blocking every bot with "GPT" or "Claude" in the name takes out the retrieval bots with the training ones.
- Trusting the file over the logs. Check status codes for each bot at the CDN.
- Treating llms.txt as access control. It is a content index, not a permission file. Validate yours with the free llms.txt generator and validator, which checks format, absolute links and broken links. Background in the llms.txt guide.
- Never revisiting. Vendors add and rename bots. Put a quarterly review in the calendar and use the AI crawler policy kit as a template.
- Blocking Googlebot to stop Gemini. Google-Extended is a separate token. Check what it does on Google's crawler list before editing Googlebot rules, and see Google AI training control in robots.txt.
How we researched this
We reviewed Google Search Central crawler documentation, Search Engine Journal reporting on publisher bot blocking and AI bot management, and recent vendor-neutral guides summarising OpenAI, Anthropic and Perplexity bot documentation. We did not run our own crawler tests. Where we relied on a secondary summary of a vendor's documentation, the text says so. All facts were checked on 2026-11-15, and bot behaviour changes often, so confirm tokens on each vendor's page before deploying.
What to do next
Open your robots.txt today, list every AI token in it, and label each as training, retrieval or user-triggered. Fix any retrieval bot you blocked by accident, then confirm in logs that it gets a 200. Record your baseline answers before you ship the change. To track the effect across engines, book a demo or start the free trial.
See what AI engines say about your brand
Free · no account · or start a 3-day trial
Questions people ask
Should I block GPTBot but allow OAI-SearchBot?
Yes, if you want to opt out of OpenAI model training and still appear in ChatGPT search. They are separate robots.txt tokens. According to GeoToolbox's summary of OpenAI's documentation, sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, while blocking GPTBot leaves that visibility intact.
What is the difference between AI training and retrieval crawlers?
A training crawler collects pages that may be used to build future models. A retrieval crawler indexes pages so an AI search product can find and cite them in answers. Blocking training is a licensing choice, while blocking retrieval can remove you from that engine's answers.
Do ChatGPT-User and Perplexity-User obey robots.txt?
Generally not reliably. They are user-triggered fetchers that load a page because a person asked, and Google's documentation says user-triggered fetchers ignore robots.txt. GeoToolbox reports that Perplexity says Perplexity-User generally ignores it. Use authentication for anything that must stay private.
Will blocking Google-Extended remove me from Google Search?
No. Google-Extended is a separate robots.txt token listed among Google's common crawlers, and GeoToolbox describes it as controlling use for Gemini training and grounding, not a crawler itself. Googlebot is a different token. Check Google's crawler documentation before changing either.
Can my CDN block AI bots that robots.txt allows?
Yes. Robots.txt is voluntary, and Search Engine Journal's Chris Green notes that CDNs, web application firewalls and bot mitigation services enforce access and can contradict your file. Check server logs for the status codes each bot receives.
How do I check if my AI bot policy changed my citations?
Record which engines name and cite you for a fixed set of buyer questions before the change, alter one bot group, confirm access in logs, then re-run the same questions after a few weeks. Compare engines separately, because answers vary between runs.
