# How to Track AI Crawlers: Logs, Analytics and Tools

> AI crawler analytics guide: find GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in server and CDN logs, spot CDN blocks and fix them today.

- Author: Sheetal Shakshi, Founder's Office, Nurdd
- Published: 2026-11-13
- Publisher: Noma by Nurdd (https://nomahq.app)
- Original page: https://nomahq.app/blog/ai-crawler-analytics-track-bots

To see which AI crawlers visit your website, search your server and CDN access logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User, then group requests by status code. A CDN can block them through bot rules even when robots.txt allows them, so a 403 or 429 is the signal.

That count is the core of AI crawler analytics. This guide gives you the commands, a worked example, a status-code table and a CDN-move checklist. Read the limits section too: we have not run these commands on a live CDN migration, and we say where that matters.

## Key takeaways

- Count requests per bot and per status code. A bot that appears only with 403, 429 or 503 is being blocked, not ignoring you.
- OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Treat every token as its own switch.
- Check the CDN or firewall layer before robots.txt. A robots file is a request to polite crawlers, not access control.
- A user-agent string can be faked. Verify any bot you act on against the vendor's published IP ranges.
- After any CDN, DNS or hosting move, re-run the same log query within 48 hours and compare it with the week before.
- Logs show who fetched a page. They do not show whether an answer named you, so pair them with prompt tracking.

## Which AI crawlers should you look for?

Start with the tokens below. Vendors add and rename bots, so confirm each one on the vendor's own page before you write a rule. For an outside view of whether these bots can reach you, the [free AI crawler checker](https://nomahq.app/tools/ai-crawler-checker) tests GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google AI against your site and generates a fixed robots.txt.

| Token to search | Operator | Role, from the vendor's page | Source |
| --- | --- | --- | --- |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | OpenAI crawler overview |
| GPTBot | OpenAI | Disallowing it signals that crawled content should not be used to train OpenAI's generative foundation models | OpenAI crawler overview |
| ChatGPT-User | OpenAI | Certain user actions in ChatGPT and Custom GPTs; not used to decide whether content appears in Search | OpenAI crawler overview |
| ClaudeBot | Anthropic | Restricting it signals that future material should be excluded from AI model training datasets | Anthropic support article |
| Claude-User | Anthropic | Supports Claude users; disabling it stops retrieval in response to a user query and may reduce visibility for user-directed search | Anthropic support article |
| Claude-SearchBot | Anthropic | Navigates the web to improve search result quality | Anthropic support article |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results; not used to crawl for foundation models | Perplexity crawler documentation |
| Perplexity-User | Perplexity | Visits a page to answer a user's question; generally ignores robots.txt | Perplexity crawler documentation |

Roles above were read on [OpenAI Developers, Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots), [Anthropic's support article on its crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Perplexity's crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), as checked on 2026-11-12. OpenAI's page says each setting is independent, so allowing GPTBot does not allow OAI-SearchBot. Its page says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

> "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." (OpenAI Developers, Overview of OpenAI Crawlers, checked 2026-11-12)

Anthropic's page also states that its bots honour robots.txt, and that if your file names ClaudeBot, Anthropic follows that rule over any general setting.

## How do I see which AI crawlers visit my website?

You have three data sources. Use whichever you can reach today, and more than one if the first shows nothing.

1. **Origin server logs** (Nginx, Apache, IIS, or your host's log download). Complete if no CDN sits in front, partial if one does.
2. **CDN or WAF logs** (request logs or security events in your CDN dashboard). These see requests before the origin does, including the ones the CDN blocked.
3. **Web analytics** (GA4 and similar). Tag-based tools rarely see crawlers, because bots generally do not run JavaScript. Absence here proves nothing.

### Step 1: pull a clean window of logs

Export at least 14 days. Crawl visits are bursty, and a 24-hour sample can miss a bot that returns weekly. If your host keeps only a few days, start saving rotated logs now.

### Step 2: count requests per bot

On a combined-format Nginx or Apache log, this command lists hits per AI token:

- `grep -Eio 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | sort | uniq -c | sort -rn`

The `-i` flag matters because user-agent casing varies. If the output is empty, either no bot visited or the log you are reading is an origin log behind a CDN that never forwarded the request. Do not conclude absence until you have checked the CDN layer in step 4.

### Step 3: split by status code

In the combined format, the status is the ninth whitespace-separated field:

- `grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | awk '{print $9}' | sort | uniq -c`

| Status pattern for one bot | Likely meaning | Next action |
| --- | --- | --- |
| Mostly 200 | Bot is reaching content | Check it reaches the pages you want cited |
| Mostly 403 | Firewall, bot-management rule or host security blocks it | Find the rule in the CDN or WAF and add an exception |
| Mostly 429 | Rate limiting treats the bot as abusive | Raise the limit for verified bot IPs |
| Mostly 503 or other 5xx | Origin overloaded or a challenge page served | Check origin capacity and any JavaScript challenge |
| Mostly 301 or 302 to one URL | Redirect chain or geo-redirect after a migration | Test the redirect with the bot's user-agent |
| Only robots.txt requests, then nothing | Bot read the file and found a Disallow | Review the matching User-agent group |
| No rows at all | No visit, or logs incomplete | Check CDN logs, then robots.txt and the sitemap |

### A worked example you can follow

The lines below are illustrative: we wrote them to show the method, and they are not real traffic. Imagine a site that moved behind a new CDN last week. Step 2 returns this tally:

| Token | Hits (14 days) | 200 | 403 | 429 |
| --- | --- | --- | --- | --- |
| OAI-SearchBot | 120 | 0 | 120 | 0 |
| GPTBot | 300 | 290 | 0 | 10 |
| ClaudeBot | 200 | 200 | 0 | 0 |
| PerplexityBot | 40 | 5 | 0 | 35 |

Read it row by row. GPTBot looks healthy, so a glance at total AI hits would say all is well. OAI-SearchBot is the problem: 120 of 120 requests refused, and per OpenAI's page a site opted out of that bot is not shown in ChatGPT search answers. PerplexityBot is mostly rate limited. The fix is two exceptions, not a robots.txt change: one for verified OpenAI search IPs, one raised rate limit for verified Perplexity IPs. The takeaway is that the per-bot status split finds faults a total never will.

### Step 4: check the CDN layer

Open the CDN's request or security log and filter by the same tokens. Look for any action other than allow: block, managed challenge, rate limit. Many CDNs ship bot-management features that can be on by default, by plan or by a one-click setting, so a rule you never wrote can be the cause. Read your provider's current bot settings page, because defaults differ by vendor and change.

### Step 5: verify the bot is real

Anyone can send a request with the user-agent GPTBot. OpenAI's page links published IP lists for its bots, and Perplexity publishes ranges for PerplexityBot and Perplexity-User and tells site owners to match each user-agent against them in a WAF rule. Do the same for any vendor that publishes ranges. Allowing a user-agent string alone opens a door for scrapers.

### Step 6: save the result as a baseline

Record the date, the window, hits per bot, the share of 200s and the top 10 URLs each bot fetched. Crawl volumes also shift across the wider web for reasons that have nothing to do with your site. A saved baseline lets you tell a change in your setup from a change in the wider crawl mix.

## Can a CDN block AI crawlers like GPTBot?

Yes. A CDN sits between the crawler and your origin, so any bot rule, rate limit, managed challenge or country filter can answer the crawler before your server does. Your robots.txt can say Allow while the CDN returns 403.

The standard itself says robots rules are requests. [RFC 9309, the Robots Exclusion Protocol](https://www.rfc-editor.org/info/rfc9309/), describes them as not a form of access authorization. Only a firewall rule actually refuses a request.

> "These rules are not a form of access authorization." (RFC 9309, Robots Exclusion Protocol)

### What Google says about CDNs and crawling

Google's Search Central post [Crawling December: CDNs and crawling, December 2024](https://developers.google.com/search/blog/2024/12/crawling-december-cdns) says Googlebot's throttling threshold is much higher when it detects that a site is backed by a CDN, so it sends more simultaneous requests. That is the upside. The risk is in the setup. [SALT.agency](https://salt.agency/blog/an-introduction-to-cdns-and-seo-benefits/) summarises the same guidance as warning that excessive blocks and challenge pages can slow crawling. The post is about Googlebot, but the mechanics apply to any crawler: the CDN decides what reaches your origin.

### Defaults can change without you touching anything

Some CDNs block AI bots out of the box. In its blog post [Content Independence Day, one year on](https://blog.cloudflare.com/agentic-internet-bot-report), Cloudflare writes that for all new domains on its network, AI training crawlers would be blocked by default unless domain owners chose otherwise. That default applies to new domains, so a site added to Cloudflare in a migration may start with training bots blocked even if the same site was open before. Cloudflare's wording covers training crawlers; whether search or user-fetch bots are affected depends on your own bot settings, so read the dashboard rather than assume.

### How a CDN move blocks crawlers

Moving to a new CDN, or switching one on for the first time, changes several things at once. Each can hide AI crawlers without a single error in your own code.

| What changes in the move | How it blocks crawlers | Test |
| --- | --- | --- |
| New default bot or security rules | Unknown bots are challenged or blocked | Request a page with a bot user-agent and read the status |
| Rate limits copied or reset | Crawl bursts hit 429 | Count 429s per bot in CDN logs |
| DNS and TLS cut-over | Some requests reach the old host, some the new, and logs split | Compare both logs for the same days |
| Redirect and cache rules | Bots get redirected, or served a cached challenge page | Fetch with the bot user-agent and follow redirects |
| Logging scope | The new origin log no longer shows blocked requests | Turn on CDN request logs before the move |

### A before-and-after test you can run for free

The week before a move, save the step 2 and step 3 counts. After the cut-over, repeat both on the same weekdays. Then run a manual check with a spoofed user-agent:

- `curl -sI -o /dev/null -w '%{http_code}\n' -A 'OAI-SearchBot' https://example.com/`
- Repeat with `-A 'GPTBot'`, `-A 'ClaudeBot'` and `-A 'PerplexityBot'`, and again on `https://example.com/robots.txt`.

| curl result | What it tells you |
| --- | --- |
| 403, 429 or a challenge page | A rule matches on user-agent. A strong lead; find the rule. |
| 200 for all tokens | No user-agent rule matched your request. The real bot comes from vendor IPs, and a rule may match IP or behaviour, so it can still be blocked. Check the logs. |
| 200 for the browser, 403 for bot tokens | Clear user-agent rule, easiest to fix. |

We have not run this against a live migration, so treat curl as a lead generator and the logs as the evidence. Our guide to a [blocked ChatGPT agent](https://nomahq.app/blog/chatgpt-agent-blocked-site) covers the same fault from the ChatGPT side.

## How do I fix a block without opening the site to every bot?

1. Decide the policy per token. Training bots and search or user-fetch bots are separate choices. The [AI Crawler Policy Kit](https://nomahq.app/resources/ai-crawler-policy-kit) sets out that decision.
2. Write explicit robots.txt groups for each token you want to allow, so a wildcard rule does not catch them.
3. In the CDN or WAF, add an exception for the verified IP ranges of the bots you allow, placed above any generic bot-blocking rule.
4. Keep rate limits for unverified traffic.
5. Wait, then re-run step 3. Perplexity's documentation says changes to its settings can take up to 24 hours to take effect, so allow at least a day before judging.

Allowing a crawler makes you eligible, not cited. If pages render only through client-side JavaScript, the crawler may fetch an empty shell. The post on [Next.js content missing from indexed HTML](https://nomahq.app/blog/nextjs-content-not-showing-in-indexed-html) covers that fault, and [technical SEO for AI search](https://nomahq.app/blog/technical-seo-for-ai-search) lists the wider checks.

## What do logs not tell you?

- **No answer data.** A GPTBot hit does not mean ChatGPT will name you. Training crawls and search crawls are different jobs.
- **No referral data.** A user-triggered fetch shows a page was opened, not that a visitor clicked through. See [measuring AI influence beyond referrals](https://nomahq.app/blog/geo-referral-traffic-measurement).
- **Spoofing.** Unverified user-agents inflate counts.
- **Sampling and retention.** A CDN may sample logs or keep them briefly. Check retention before you rely on a month of history.

## Which tools track AI crawlers, and when do you need one?

| Method | Cost | Shows | Gap |
| --- | --- | --- | --- |
| Command line on raw logs | Free | Hits and status per bot | Manual, no trend, no IP verification |
| CDN or WAF dashboard | Depends on your plan | Blocked and allowed requests at the edge | Bot lists differ by vendor; no AI answer context |
| Logs in a spreadsheet or warehouse | Your time | Trends per URL and bot | You maintain the user-agent list |
| Agent analytics that reads server logs | Varies by tool | Named agent per request, per page, over time | Needs your server logs |

The command-line route is enough for a one-off audit. It stops being enough when you want a weekly trend or per-page views without rewriting the script. For the concept, read [what agent analytics is](https://nomahq.app/blog/what-is-agent-analytics).

## Where does Noma fit?

Noma by Nurdd is an AEO (answer engine optimisation) and AI visibility platform. Its [agent analytics](https://nomahq.app/capabilities/agent-analytics) shows which AI agents, such as GPTBot, ClaudeBot and PerplexityBot, read your site, from your server logs, and names the agent behind each request. Noma reports which AI crawlers fetch each page and which assistants send visitors back. It also asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, your buyers' questions daily.

See it on your own brand: [book a demo](https://nomahq.app/demo) or [start the 3-day free trial](https://platform.nomahq.app/signup). A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged.

## A checklist for your next CDN or hosting change

1. Export 14 days of logs and save per-bot counts and status splits.
2. Turn on CDN request logging before the cut-over.
3. List every bot rule, rate limit and challenge setting in the new setup, including provider defaults for new domains.
4. Add exceptions for verified IP ranges of the bots you allow.
5. Fetch robots.txt and key pages with bot user-agents after the change.
6. Compare counts 48 hours later, then again after one week.
7. Check that your llms.txt still loads and its links resolve. The [free llms.txt generator and validator](https://nomahq.app/tools/llms-txt) checks the file and every link in it.

## How we researched this

We reviewed vendor documentation (OpenAI's crawler overview, Anthropic's support article, Perplexity's crawler page, Cloudflare's blog), Google Search Central's CDN post, the RFC 9309 standard and one agency explainer on CDNs. Facts and Noma details were checked on 2026-11-12. We ran no log tests of our own: the worked example is illustrative, and the curl procedure is a method we describe, not one we tested on a live migration.

## What to do next

Run steps 2 and 3 on 14 days of logs today and note any bot that shows only 403, 429 or 5xx responses. Then compare crawler access with what AI engines say about you: [book a demo](https://nomahq.app/demo) or [start the free trial](https://platform.nomahq.app/signup). You can also read our [guide to AI agent website access](https://nomahq.app/blog/ai-agent-website-access).
