Noma by Nurdd (NomaHQ)Noma
Sign up

Free kit · PDF

Robots.txt for AI bots: which AI crawlers to allow, block and verify

A working kit for setting an AI crawler policy: the three kinds of AI bot and what blocking each one costs, a per-bot decision matrix, robots.txt templates for three stances, CDN and WAF checks, a server-log worksheet, an agent-readiness checklist and a one-page brief for IT.

  • Per-bot decision matrix covering OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon and others
  • Robots.txt templates for three stances: open, search-yes-training-no, closed
  • Server-log review worksheet with the fields to pull and the checks to run
  • Agent-readiness checklist for price, stock, booking and demo requests
  • One-page brief for IT and security, plus a quarterly review checklist

Get your free copy

Instant download, plus a copy by email. No spam: one email with your link, and a reply from a person if you write back.

The short answer

Blocking GPTBot only keeps your pages out of OpenAI's model training; it does not remove you from ChatGPT search, which uses OAI-SearchBot, or from pages ChatGPT opens for a user, which use ChatGPT-User. If you want to be cited, allow search and user bots, decide training bots separately, and check your CDN is not blocking them.

For technical SEO leads, web and platform engineers, security and IT teams, and e-commerce leads

What you will learn

  1. Chapter 1

    The three kinds of AI crawler

    • AI companies run separate bots for training, search indexing and user-triggered fetches, and robots.txt treats each token independently.
    • Blocking a training crawler keeps you out of future training data; it does not remove you from today's answers.
    • Blocking a search indexer, or Googlebot, removes you from that assistant's cited answers.
    • User-triggered fetchers may ignore robots.txt by the vendors' own account, so control them at the CDN or firewall.
    • Verify a bot by IP range, DNS or signature before trusting its user agent.

  2. Chapter 2

    Decide bot by bot

    • Write a reason next to each bot decision, so the policy survives the next person who questions it.
    • Search indexers and user fetchers decide whether you are cited; training crawlers decide whether you train future models.
    • The search-yes-training-no stance is the common middle ground, but disallowing Google-Extended can also affect grounding in Gemini Apps.
    • A named robots.txt group replaces the catch-all group for that bot, so repeat private-path rules inside it.
    • Every host needs its own robots.txt, and a 5xx on that file can pause crawling.

  3. Chapter 3

    The layer above robots.txt: CDN, WAF and bot management

    • When robots.txt and the edge disagree, the edge wins, and the block is invisible from the robots.txt file.
    • Cloudflare changed its AI bot defaults for new domains on 15 September 2026; check which policy your zone is on.
    • Managed robots.txt can prepend rules you did not write, so always read the live file.
    • Akamai now separates training, search and agent categories; check how older single-category rules were mapped.
    • JavaScript challenges and rate limits block AI crawlers as effectively as an explicit rule.

  4. Chapter 4

    Prove what the bots actually reach

    • Read edge logs before origin logs, because blocked requests never reach the origin.
    • Separate verified from unverified requests before you count anything.
    • Track success rate, coverage and freshness per bot for your priority URLs.
    • Assume AI crawlers do not run JavaScript: put the facts you want cited in the raw HTML.
    • Check X-Robots-Tag headers as well as page markup, on every host, after every major deploy.

  5. Chapter 5

    llms.txt, Markdown copies and agent readiness

    • llms.txt is a proposed curated guide to a site; Google says its AI search features do not need it, and no answer engine has documented using it to choose citations.
    • Publish llms.txt and Markdown copies only if you can keep them accurate, and generate them from the same source as your pages.
    • Put prices, availability and plan details in server-rendered HTML and in structured data, where agents and crawlers can both read them.
    • Test the tasks a buyer might delegate (price, stock, booking, demo) and fix what breaks them.
    • Accessible forms and labelled controls are what make a site usable by agents.

  6. Chapter 6

    Brief IT, then keep the policy true

    • Write down an owner for robots.txt, the CDN and WAF, rendering and the legal position.
    • Ask IT for specific, verifiable, reversible changes: named bots, named rules, a rate ceiling and a rollback.
    • Monitor the live robots.txt and the bots' status codes, not just the file in the repository.
    • Review the register, the logs and the decisions every quarter, because vendors and CDN defaults change.

7 worksheets inside

The chapters explain the method. The worksheets are what you fill in to run it: print them, or copy the tables into a document of your own.

  • W1Per-bot decision matrix
  • W2Robots.txt templates for three stances
  • W3CDN and WAF access check
  • W4Server-log review
  • W5Agent-readiness checklist
  • W6One-page brief for IT and security
  • W7Quarterly review checklist
Get the kit

Questions

Should I block GPTBot?
Only if you do not want your pages used to train OpenAI's future models. OpenAI says disallowing GPTBot signals exactly that, and it has no bearing on ChatGPT search, which uses OAI-SearchBot. You can block GPTBot and allow OAI-SearchBot in the same file.
What is the difference between OAI-SearchBot and GPTBot?
OAI-SearchBot surfaces websites in ChatGPT's search features; OpenAI says sites that opt out of it will not be shown in ChatGPT search answers. GPTBot collects content that may be used to train OpenAI's foundation models. Each is controlled separately in robots.txt.
How do I allow ChatGPT to crawl my site?
Make sure robots.txt does not disallow OAI-SearchBot or ChatGPT-User, then check your CDN and firewall are not blocking or challenging them. OpenAI publishes IP ranges for each bot, so your security team can allow the verified bots without opening the door to impostors. OpenAI says robots.txt changes take about 24 hours to take effect.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode use the ordinary Search index built by Googlebot. Google-Extended controls Gemini model training and grounding in Gemini Apps and Vertex AI.
Do AI crawlers obey robots.txt?
Training crawlers and search indexers from the major vendors say they do. User-triggered fetchers are different: OpenAI, Perplexity, Meta, Amazon and Google each say theirs may not follow robots.txt, because a person requested the page. To stop those, use a rule at your CDN or firewall.
Do I need an llms.txt file?
Not for Google: its documentation says AI features in Search need no special AI text files. No answer engine has documented using llms.txt to choose what to cite. Chrome's Lighthouse now checks for one in an experimental agentic browsing category, so publish one if you can keep it accurate, without expecting it to change citations.
Why is my site missing from AI answers when robots.txt allows the bots?
Check the edge first: a CDN or WAF rule, a bot challenge or a rate limit that blocks the crawler before robots.txt matters. Other causes are content that only appears after JavaScript runs, a noindex in a response header, or a robots.txt returning an error. Check edge logs for the bot's status codes first.
Is my site ready for AI agents?
Test the tasks a buyer might delegate: finding a price, checking stock, booking, requesting a demo. If the facts are in server-rendered HTML, forms have labelled fields and verified fetchers are not challenged, an agent has a fair chance. Worksheet 5 in this kit is a task-by-task checklist.
Is The AI Crawler Policy Kit free?
Yes. Give a first name and a work email and the download starts at once. We also email you the link, and one person from the Noma team follows up a day later; reply and say so if you would rather not hear from us.
Why do you ask for a work email?
The playbooks are written for teams at companies, and a work address tells us which company is reading, so a follow-up can be about your brand rather than generic. Personal and temporary inboxes are not accepted.
What format is it?
A PDF of 46 A4 pages, laid out to print, with worksheets you can fill in by hand or copy into a document of your own.

More playbooks

Updated 20 October 2026 · Written by the Noma team · All resources