Noma by Nurdd (NomaHQ)Noma
technical SEO

How to Track AI Crawlers: Logs, Analytics and Tools

AI crawler analytics guide: find GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot in server and CDN logs, spot CDN blocks and fix them today.

Sheetal Shakshi11 min read

To see which AI crawlers visit your website, search your server and CDN access logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User, then group requests by status code. A CDN can block them through bot rules even when robots.txt allows them, so a 403 or 429 is the signal.

That count is the core of AI crawler analytics. This guide gives you the commands, a worked example, a status-code table and a CDN-move checklist. Read the limits section too: we have not run these commands on a live CDN migration, and we say where that matters.

Key takeaways

  • Count requests per bot and per status code. A bot that appears only with 403, 429 or 503 is being blocked, not ignoring you.
  • OpenAI, Anthropic and Perplexity each run separate bots for training, search and user-triggered fetches. Treat every token as its own switch.
  • Check the CDN or firewall layer before robots.txt. A robots file is a request to polite crawlers, not access control.
  • A user-agent string can be faked. Verify any bot you act on against the vendor's published IP ranges.
  • After any CDN, DNS or hosting move, re-run the same log query within 48 hours and compare it with the week before.
  • Logs show who fetched a page. They do not show whether an answer named you, so pair them with prompt tracking.

Which AI crawlers should you look for?

Start with the tokens below. Vendors add and rename bots, so confirm each one on the vendor's own page before you write a rule. For an outside view of whether these bots can reach you, the free AI crawler checker tests GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google AI against your site and generates a fixed robots.txt.

Token to searchOperatorRole, from the vendor's pageSource
OAI-SearchBotOpenAISurfaces websites in ChatGPT's search featuresOpenAI crawler overview
GPTBotOpenAIDisallowing it signals that crawled content should not be used to train OpenAI's generative foundation modelsOpenAI crawler overview
ChatGPT-UserOpenAICertain user actions in ChatGPT and Custom GPTs; not used to decide whether content appears in SearchOpenAI crawler overview
ClaudeBotAnthropicRestricting it signals that future material should be excluded from AI model training datasetsAnthropic support article
Claude-UserAnthropicSupports Claude users; disabling it stops retrieval in response to a user query and may reduce visibility for user-directed searchAnthropic support article
Claude-SearchBotAnthropicNavigates the web to improve search result qualityAnthropic support article
PerplexityBotPerplexitySurfaces and links sites in Perplexity search results; not used to crawl for foundation modelsPerplexity crawler documentation
Perplexity-UserPerplexityVisits a page to answer a user's question; generally ignores robots.txtPerplexity crawler documentation

Roles above were read on OpenAI Developers, Overview of OpenAI Crawlers, Anthropic's support article on its crawlers and Perplexity's crawler documentation, as checked on 2026-11-12. OpenAI's page says each setting is independent, so allowing GPTBot does not allow OAI-SearchBot. Its page says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." (OpenAI Developers, Overview of OpenAI Crawlers, checked 2026-11-12)

Anthropic's page also states that its bots honour robots.txt, and that if your file names ClaudeBot, Anthropic follows that rule over any general setting.

How do I see which AI crawlers visit my website?

You have three data sources. Use whichever you can reach today, and more than one if the first shows nothing.

  1. Origin server logs (Nginx, Apache, IIS, or your host's log download). Complete if no CDN sits in front, partial if one does.
  2. CDN or WAF logs (request logs or security events in your CDN dashboard). These see requests before the origin does, including the ones the CDN blocked.
  3. Web analytics (GA4 and similar). Tag-based tools rarely see crawlers, because bots generally do not run JavaScript. Absence here proves nothing.

Step 1: pull a clean window of logs

Export at least 14 days. Crawl visits are bursty, and a 24-hour sample can miss a bot that returns weekly. If your host keeps only a few days, start saving rotated logs now.

Step 2: count requests per bot

On a combined-format Nginx or Apache log, this command lists hits per AI token:

  • grep -Eio 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | sort | uniq -c | sort -rn

The -i flag matters because user-agent casing varies. If the output is empty, either no bot visited or the log you are reading is an origin log behind a CDN that never forwarded the request. Do not conclude absence until you have checked the CDN layer in step 4.

Step 3: split by status code

In the combined format, the status is the ninth whitespace-separated field:

  • grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | awk '{print $9}' | sort | uniq -c
Status pattern for one botLikely meaningNext action
Mostly 200Bot is reaching contentCheck it reaches the pages you want cited
Mostly 403Firewall, bot-management rule or host security blocks itFind the rule in the CDN or WAF and add an exception
Mostly 429Rate limiting treats the bot as abusiveRaise the limit for verified bot IPs
Mostly 503 or other 5xxOrigin overloaded or a challenge page servedCheck origin capacity and any JavaScript challenge
Mostly 301 or 302 to one URLRedirect chain or geo-redirect after a migrationTest the redirect with the bot's user-agent
Only robots.txt requests, then nothingBot read the file and found a DisallowReview the matching User-agent group
No rows at allNo visit, or logs incompleteCheck CDN logs, then robots.txt and the sitemap

A worked example you can follow

The lines below are illustrative: we wrote them to show the method, and they are not real traffic. Imagine a site that moved behind a new CDN last week. Step 2 returns this tally:

TokenHits (14 days)200403429
OAI-SearchBot12001200
GPTBot300290010
ClaudeBot20020000
PerplexityBot405035

Read it row by row. GPTBot looks healthy, so a glance at total AI hits would say all is well. OAI-SearchBot is the problem: 120 of 120 requests refused, and per OpenAI's page a site opted out of that bot is not shown in ChatGPT search answers. PerplexityBot is mostly rate limited. The fix is two exceptions, not a robots.txt change: one for verified OpenAI search IPs, one raised rate limit for verified Perplexity IPs. The takeaway is that the per-bot status split finds faults a total never will.

Step 4: check the CDN layer

Open the CDN's request or security log and filter by the same tokens. Look for any action other than allow: block, managed challenge, rate limit. Many CDNs ship bot-management features that can be on by default, by plan or by a one-click setting, so a rule you never wrote can be the cause. Read your provider's current bot settings page, because defaults differ by vendor and change.

Step 5: verify the bot is real

Anyone can send a request with the user-agent GPTBot. OpenAI's page links published IP lists for its bots, and Perplexity publishes ranges for PerplexityBot and Perplexity-User and tells site owners to match each user-agent against them in a WAF rule. Do the same for any vendor that publishes ranges. Allowing a user-agent string alone opens a door for scrapers.

Step 6: save the result as a baseline

Record the date, the window, hits per bot, the share of 200s and the top 10 URLs each bot fetched. Crawl volumes also shift across the wider web for reasons that have nothing to do with your site. A saved baseline lets you tell a change in your setup from a change in the wider crawl mix.

Can a CDN block AI crawlers like GPTBot?

Yes. A CDN sits between the crawler and your origin, so any bot rule, rate limit, managed challenge or country filter can answer the crawler before your server does. Your robots.txt can say Allow while the CDN returns 403.

The standard itself says robots rules are requests. RFC 9309, the Robots Exclusion Protocol, describes them as not a form of access authorization. Only a firewall rule actually refuses a request.

"These rules are not a form of access authorization." (RFC 9309, Robots Exclusion Protocol)

What Google says about CDNs and crawling

Google's Search Central post Crawling December: CDNs and crawling, December 2024 says Googlebot's throttling threshold is much higher when it detects that a site is backed by a CDN, so it sends more simultaneous requests. That is the upside. The risk is in the setup. SALT.agency summarises the same guidance as warning that excessive blocks and challenge pages can slow crawling. The post is about Googlebot, but the mechanics apply to any crawler: the CDN decides what reaches your origin.

Defaults can change without you touching anything

Some CDNs block AI bots out of the box. In its blog post Content Independence Day, one year on, Cloudflare writes that for all new domains on its network, AI training crawlers would be blocked by default unless domain owners chose otherwise. That default applies to new domains, so a site added to Cloudflare in a migration may start with training bots blocked even if the same site was open before. Cloudflare's wording covers training crawlers; whether search or user-fetch bots are affected depends on your own bot settings, so read the dashboard rather than assume.

How a CDN move blocks crawlers

Moving to a new CDN, or switching one on for the first time, changes several things at once. Each can hide AI crawlers without a single error in your own code.

What changes in the moveHow it blocks crawlersTest
New default bot or security rulesUnknown bots are challenged or blockedRequest a page with a bot user-agent and read the status
Rate limits copied or resetCrawl bursts hit 429Count 429s per bot in CDN logs
DNS and TLS cut-overSome requests reach the old host, some the new, and logs splitCompare both logs for the same days
Redirect and cache rulesBots get redirected, or served a cached challenge pageFetch with the bot user-agent and follow redirects
Logging scopeThe new origin log no longer shows blocked requestsTurn on CDN request logs before the move

A before-and-after test you can run for free

The week before a move, save the step 2 and step 3 counts. After the cut-over, repeat both on the same weekdays. Then run a manual check with a spoofed user-agent:

  • curl -sI -o /dev/null -w '%{http_code}\n' -A 'OAI-SearchBot' https://example.com/
  • Repeat with -A 'GPTBot', -A 'ClaudeBot' and -A 'PerplexityBot', and again on https://example.com/robots.txt.
curl resultWhat it tells you
403, 429 or a challenge pageA rule matches on user-agent. A strong lead; find the rule.
200 for all tokensNo user-agent rule matched your request. The real bot comes from vendor IPs, and a rule may match IP or behaviour, so it can still be blocked. Check the logs.
200 for the browser, 403 for bot tokensClear user-agent rule, easiest to fix.

We have not run this against a live migration, so treat curl as a lead generator and the logs as the evidence. Our guide to a blocked ChatGPT agent covers the same fault from the ChatGPT side.

How do I fix a block without opening the site to every bot?

  1. Decide the policy per token. Training bots and search or user-fetch bots are separate choices. The AI Crawler Policy Kit sets out that decision.
  2. Write explicit robots.txt groups for each token you want to allow, so a wildcard rule does not catch them.
  3. In the CDN or WAF, add an exception for the verified IP ranges of the bots you allow, placed above any generic bot-blocking rule.
  4. Keep rate limits for unverified traffic.
  5. Wait, then re-run step 3. Perplexity's documentation says changes to its settings can take up to 24 hours to take effect, so allow at least a day before judging.

Allowing a crawler makes you eligible, not cited. If pages render only through client-side JavaScript, the crawler may fetch an empty shell. The post on Next.js content missing from indexed HTML covers that fault, and technical SEO for AI search lists the wider checks.

What do logs not tell you?

  • No answer data. A GPTBot hit does not mean ChatGPT will name you. Training crawls and search crawls are different jobs.
  • No referral data. A user-triggered fetch shows a page was opened, not that a visitor clicked through. See measuring AI influence beyond referrals.
  • Spoofing. Unverified user-agents inflate counts.
  • Sampling and retention. A CDN may sample logs or keep them briefly. Check retention before you rely on a month of history.

Which tools track AI crawlers, and when do you need one?

MethodCostShowsGap
Command line on raw logsFreeHits and status per botManual, no trend, no IP verification
CDN or WAF dashboardDepends on your planBlocked and allowed requests at the edgeBot lists differ by vendor; no AI answer context
Logs in a spreadsheet or warehouseYour timeTrends per URL and botYou maintain the user-agent list
Agent analytics that reads server logsVaries by toolNamed agent per request, per page, over timeNeeds your server logs

The command-line route is enough for a one-off audit. It stops being enough when you want a weekly trend or per-page views without rewriting the script. For the concept, read what agent analytics is.

Where does Noma fit?

Noma by Nurdd is an AEO (answer engine optimisation) and AI visibility platform. Its agent analytics shows which AI agents, such as GPTBot, ClaudeBot and PerplexityBot, read your site, from your server logs, and names the agent behind each request. Noma reports which AI crawlers fetch each page and which assistants send visitors back. It also asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, your buyers' questions daily.

See it on your own brand: book a demo or start the 3-day free trial. A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged.

A checklist for your next CDN or hosting change

  1. Export 14 days of logs and save per-bot counts and status splits.
  2. Turn on CDN request logging before the cut-over.
  3. List every bot rule, rate limit and challenge setting in the new setup, including provider defaults for new domains.
  4. Add exceptions for verified IP ranges of the bots you allow.
  5. Fetch robots.txt and key pages with bot user-agents after the change.
  6. Compare counts 48 hours later, then again after one week.
  7. Check that your llms.txt still loads and its links resolve. The free llms.txt generator and validator checks the file and every link in it.

How we researched this

We reviewed vendor documentation (OpenAI's crawler overview, Anthropic's support article, Perplexity's crawler page, Cloudflare's blog), Google Search Central's CDN post, the RFC 9309 standard and one agency explainer on CDNs. Facts and Noma details were checked on 2026-11-12. We ran no log tests of our own: the worked example is illustrative, and the curl procedure is a method we describe, not one we tested on a live migration.

What to do next

Run steps 2 and 3 on 14 days of logs today and note any bot that shows only 403, 429 or 5xx responses. Then compare crawler access with what AI engines say about you: book a demo or start the free trial. You can also read our guide to AI agent website access.

See what AI engines say about your brand

Free · no account · or start a 3-day trial

Questions people ask

How do I see which AI crawlers visit my website?

Search your server or CDN access logs for the user-agent tokens GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User, then count requests per bot and per status code. Web analytics tools usually miss them because bots generally do not run JavaScript. Check CDN logs as well as origin logs.

Can a CDN block AI crawlers like GPTBot?

Yes. A CDN answers requests before your origin server, so a bot rule, rate limit or challenge can return a 403, 429 or challenge page even when robots.txt allows the crawler. Cloudflare writes that for all new domains on its network, AI training crawlers would be blocked by default unless the owner chose otherwise.

Does blocking GPTBot stop ChatGPT from citing my site?

Not by itself. OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as separate settings. Its page says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

Why do my logs show no AI crawler visits?

Either none visited, or you are reading the wrong layer. A CDN may answer or block requests before they reach your origin log. Check CDN request logs, widen the window to 14 days, and make sure robots.txt and the sitemap are reachable.

Can I trust the user-agent string in my logs?

No, it can be faked. Compare the source IP with the ranges the vendor publishes, as OpenAI and Perplexity do for their bots. Prefer allowing by verified IP rather than by user-agent alone.

Does a crawler visit mean AI engines will name my brand?

No. A crawl shows access, not an answer. To know whether engines name you, track a fixed set of buyer prompts across engines and compare that with your crawl data.

← Back to the blog