Free AI Crawler Checker
See if ChatGPT, Claude, Perplexity and Google AI can read your site, and fix your robots.txt in one click.
- Free, no signup
- 30 AI bots checked
- Fixed robots.txt in one click
What this AI crawler checker tests
It reads your robots.txt the way RFC 9309 says a crawler must, and works out, for each of 30 AI bots, whether it may read your homepage, a path you choose and common sections such as /blog and /pricing. It then requests your homepage as a browser and as three AI bots to see whether a firewall treats them differently, and reads your homepage's meta robots tags and X-Robots-Tag header. A bot is matched only by its exact token, so a Googlebot rule does not catch Googlebot-Image and a GPTBot rule does not catch ChatGPT-User.
Search bots, user fetchers and training bots
AI companies run different bots for different jobs. Search bots build the index an assistant searches when it answers. User fetchers open a page right now because someone asked about it. Training bots collect pages for future models. Blocking a training bot is a choice about training, and it does not remove you from answers today. Blocking a search bot or a user fetcher does.
| Token | Company | Purpose | Blocking it costs AI answers |
|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search | Yes |
| ChatGPT-User | OpenAI | User-triggered fetch | Yes |
| GPTBot | OpenAI | Training | No |
| OAI-AdsBot | OpenAI | Other | No |
| Claude-SearchBot | Anthropic | AI search | Yes |
| Claude-User | Anthropic | User-triggered fetch | Yes |
| ClaudeBot | Anthropic | Training | No |
| PerplexityBot | Perplexity | AI search | Yes |
| Perplexity-User | Perplexity | User-triggered fetch | Yes |
| Googlebot | AI search | Yes | |
| Google-Extended | Control token | Yes | |
| Google-CloudVertexBot | Other | No | |
| Google-Agent | User-triggered fetch | Yes | |
| Google-GeminiNotebook | User-triggered fetch | Yes | |
| Google-Pinpoint | User-triggered fetch | Yes | |
| Applebot | Apple | AI search | Yes |
| Applebot-Extended | Apple | Control token | No |
| meta-externalagent | Meta | Training | No |
| meta-webindexer | Meta | AI search | Yes |
| meta-externalfetcher | Meta | User-triggered fetch | Yes |
| Amazonbot | Amazon | Training | No |
| Amzn-SearchBot | Amazon | AI search | Yes |
| Amzn-User | Amazon | User-triggered fetch | Yes |
| MistralAI-Index | Mistral | AI search | Yes |
| MistralAI-User | Mistral | User-triggered fetch | Yes |
| MistralAI-Training | Mistral | Training | No |
| Bingbot | Microsoft | AI search | Yes |
| DuckAssistBot | DuckDuckGo | User-triggered fetch | Yes |
| CCBot | Common Crawl | Open dataset | No |
| Bytespider | ByteDance | Undocumented | Not documented |
How to allow AI search but block AI training
Give the training bots a group with Disallow: / and leave the search and user bots allowed. The catch is that a bot with its own group ignores your User-agent: * rules, so a bare "Allow: /" group for OAI-SearchBot would open your admin and checkout paths to it. The generator above copies those private paths into the AI groups and checks the result before showing it, and it never touches your Googlebot, Bingbot or Applebot rules.
Firewalls and CDNs that block AI bots
robots.txt is only half of crawler access. A CDN bot setting or a web application firewall rule can turn an AI crawler away even when robots.txt invites it in, and nothing in your HTML shows it. If the checker says a bot may be blocked, look at your CDN's bot settings and your firewall rules for anything that matches AI user agents. To see whether AI engines actually mention you once they can read you, run the free AI visibility audit, or check the rest of your site with the AI SEO Checker.
Common questions
- Does blocking GPTBot remove my site from ChatGPT?
- No. OpenAI documents GPTBot as the crawler that collects content which may be used to train its models. ChatGPT's search answers use OAI-SearchBot, and ChatGPT-User visits pages when a user asks. You can block GPTBot and still appear in ChatGPT answers, as long as OAI-SearchBot and ChatGPT-User are allowed.
- Which robots.txt lines let ChatGPT, Claude and Perplexity cite my site?
- Allow OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, and PerplexityBot and Perplexity-User for Perplexity. If your robots.txt has no group for them, they follow your User-agent: * rules. If you give them a group of their own, they ignore your * rules entirely, so copy any private paths into it. The generator on this page does that for you.
- Can I block Google AI Overviews without leaving Google Search?
- Not with robots.txt. AI Overviews and AI Mode are built from Google's ordinary search index, which Googlebot crawls, so blocking Googlebot removes you from Google Search as well. Google-Extended does not affect Search or AI Overviews.
- What is Google-Extended?
- A control token, not a crawler. Google crawls with its usual user agents and reads Google-Extended only from robots.txt. Google says it controls whether your pages may be used to train Gemini models and to ground answers in Gemini Apps and the Vertex AI Gemini API. Blocking it does not affect Google Search, but it can keep you out of some Gemini answers.
- Why does the checker say my firewall may be blocking AI bots?
- We request your homepage once as a normal browser and once each as OAI-SearchBot, ClaudeBot and PerplexityBot. If a bot request gets a different status code or a challenge page, a CDN or firewall rule is probably treating AI bots differently. Some firewalls verify real bots by their published IP addresses, so our request can be refused while the real bot gets through. That is why we say a bot may be blocked, never that it is.
- How long do AI companies take to notice robots.txt changes?
- It varies by vendor. OpenAI says changes for OAI-SearchBot take about 24 hours, Perplexity says up to 24 hours, Meta may cache robots.txt for up to 24 hours, DuckDuckGo says a disallow for DuckAssistBot takes effect after 72 hours, and Amazon may use a cached copy up to 30 days old. RFC 9309, the robots.txt standard, says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.