AI crawler analytics is the practice of separating AI bot requests in your server logs by user agent and purpose, verifying each one against the operator's published IP ranges, and then comparing the pages those bots fetched with the pages AI answers later cite. Logs show access. They do not show influence, so you need both halves.
Key takeaways
- Split AI bots into three jobs: training crawlers, search-index crawlers and user-triggered fetchers. They answer different questions and need different rules.
- Treat a user-agent string as a claim. In one published week of logs, only 12 of 54 requests claiming to be PerplexityBot came from Perplexity's listed ranges.
- Report verified hits per URL per week, not raw hit totals. Raw totals overstate real crawling.
- Join crawler hits to citation data by URL. Four outcomes (crawled and cited, crawled and not cited, cited and not crawled, neither) each point to a different fix.
- Logs cannot show that a crawl changed an answer. Only repeated prompt tracking can show whether you are named and cited.
Which AI crawlers should I look for in my server logs?
Start with the bots the three largest assistants publish documentation for. According to Search Engine Journal, February 2026, Anthropic lists three bots (ClaudeBot, Claude-User and Claude-SearchBot) and OpenAI also runs separate bots for training, search and user requests. Perplexity's documentation, November 2026 describes two: PerplexityBot and Perplexity-User. Lumping them into one "AI bots" bucket hides the only distinction that matters: whether the visit could lead to you being shown to a user.
| User agent | Operator | Job | robots.txt behaviour (per vendor docs) | What a hit tells you |
|---|---|---|---|---|
| GPTBot | OpenAI | Training data collection | Has its own robots.txt tag | Content is being considered for model training, not for live answers |
| OAI-SearchBot | OpenAI | Search indexing for ChatGPT | Has its own robots.txt tag; OpenAI says to use it for search opt-outs | Page may be eligible to appear in ChatGPT search results |
| ChatGPT-User | OpenAI | User-initiated fetch | May not be governed by robots.txt in the same way | A person asked ChatGPT something and it opened your page |
| ClaudeBot | Anthropic | Training data collection | Controlled through robots.txt | Training collection only |
| Claude-SearchBot | Anthropic | Search indexing | Controlled through robots.txt | Page is being indexed for Claude search answers |
| Claude-User | Anthropic | User-initiated fetch | Controlled through robots.txt | A Claude user's question triggered a retrieval |
| PerplexityBot | Perplexity | Surfacing and linking sites in search results | Honours robots.txt; not used for foundation model training | Page is in the pool Perplexity can link to |
| Perplexity-User | Perplexity | User-initiated fetch | Generally ignores robots.txt rules | A person's Perplexity question led to a live fetch |
Sources: OpenAI bot documentation for the OpenAI robots.txt tags and published IP lists; Perplexity documentation, November 2026 for the Perplexity rows; Search Engine Journal, February 2026 for the Anthropic rows and the ChatGPT-User note, which cites Anthropic's crawler page. As checked on 2026-11-16. Vendors change these pages, so re-read them before you write rules.
Add three more to your filter: Googlebot, Bingbot and CCBot (Common Crawl). Keep them in a separate "classic and archive" group so they do not inflate your AI totals.
Why the three-job split changes what you do
Anthropic's documentation warns that blocking its search bot may reduce your visibility in search results. As Search Engine Journal, February 2026 quotes it, blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". Blocking a training bot does not carry that cost, and the vendors' separate tags mean blocking the training bot does not by itself block their search or user fetchers. If your log report shows one number for "ClaudeBot", you cannot tell whether you are exposed to the first risk or the second. For the policy side, see AI agent website access: block, allow or serve them.
How do I analyse AI crawler traffic on my website?
You need raw access logs or CDN request logs. JavaScript analytics such as a standard page-tag tool will not see most bots, because crawlers rarely run your tracking script. If you only have page-tag analytics, this is a gap, not a minor one. The agent analytics explainer covers the definition; the steps below are the setup.
- Get 30 days of logs. Pull them from your web server, load balancer or CDN. Keep the fields: timestamp, client IP, request path, status code, user agent, bytes sent and referrer.
- Filter by user-agent token. Match the tokens in the table above case-insensitively. Write each match to a column named "claimed bot".
- Verify the IP. Check each client IP against the operator's published list. OpenAI links JSON files for each of its bots on its bot page, and Perplexity publishes files for PerplexityBot and Perplexity-User. The crawlerpeek study, October 2026 notes that Anthropic, Bing, Apple and Common Crawl publish range lists too. Label each row verified or unverified.
- Group by job. Tag each verified row as training, search index or user fetch using the table.
- Aggregate by URL and week. Count verified hits per URL per job per week. Add the status code mix (200, 301, 403, 404, 5xx).
- Read three things: which sections are crawled, which are never touched, and which return errors to bots.
To check whether your robots.txt currently lets these agents in at all, run the free AI Crawler Checker. It tests whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google AI can crawl your site and generates a fixed robots.txt. Compare its verdict with what your logs show: a bot the checker says is allowed but that never appears in logs is a different problem from one that is blocked.
What to read in the logs
- Status codes per bot. A search-index bot receiving 403 or 429 from your firewall can be a silent failure. Perplexity's own documentation tells site owners with a web application firewall to explicitly allow its bots and ranges.
- Path coverage. Compare crawled paths against your sitemap. Product, pricing and comparison pages that no search-index bot fetches in 30 days deserve a technical check first.
- Fetch of /llms.txt and /robots.txt. Count which verified bots request /llms.txt. If none do, you have your own evidence on how much that file matters for your site. Validate the file itself with the free llms.txt generator and validator, which checks format, absolute links and broken links.
- Rendering. If bots receive 200 but your content is built client-side, they may fetch an empty shell. See Next.js content not in indexed HTML.
- Burst patterns. User-fetch agents arrive in small clusters tied to one question. Index crawlers arrive in steady sweeps. Mixing them hides both.
Why a user-agent string is not proof
The most useful habit in AI crawler analytics is distrust of the user-agent field. Anyone can send "GPTBot" in a request header. The crawlerpeek analysis, October 2026 checked one week of logs (19 to 25 October 2026) on four small sites against operator-published IP lists. Its results:
| Claimed bot | Requests in published ranges | Share |
|---|---|---|
| OAI-SearchBot | 2,315 of 2,356 | 98% |
| Googlebot | 1,923 of 2,003 | 96% |
| ClaudeBot | 719 of 748 | 96% |
| GPTBot | 493 of 524 | 94% |
| ChatGPT-User | 162 of 200 | 81% |
| Bingbot | 133 of 168 | 79% |
| PerplexityBot | 12 of 54 | 22% |
| CCBot | 0 of 36 | 0% |
The author is explicit about limits: one week, four small sites, and published lists can lag new addresses, so "unverified" means "not in the list", not "proven fake". A separate site owner reported on Open Source Analytics, November 2026 that AI crawlers made almost three times as many requests as search engines on their site, and that a third of the "Googlebot" requests were fake. Both are single-site observations, not benchmarks. Use them as a reason to run the check on your own logs, not as a number to quote about the web.
A worked example from the published numbers
Add the claimed requests in the crawlerpeek table: 2,356 + 2,003 + 748 + 524 + 200 + 168 + 54 + 36 = 6,089. Add the in-range requests: 2,315 + 1,923 + 719 + 493 + 162 + 133 + 12 + 0 = 5,757. The difference is 332, which matches the 332 unmatched requests the author reports. A raw count would therefore overstate real crawling by about 5.8% across the set, and PerplexityBot by roughly four and a half times (54 claimed against 12 verified). If you report raw user-agent counts to a leadership team, the bots with the weakest verification are the ones you will over-count most.
How do I link crawler visits to later AI citations?
Crawler logs sit on one side and AI answers on the other. The link is the URL. Here is a method you can run in a spreadsheet.
- List your priority URLs. Use the 20 to 50 pages that should win buyer questions: product, pricing, comparison, definition and how-to pages.
- Add verified hits by job. For each URL, columns for search-index hits and user-fetch hits over the last 28 days. Leave training hits in a separate column.
- Add citation data. Run a fixed set of buyer prompts across engines on a schedule and record which URLs are cited. Fixed wording matters: AI answers change from run to run, so one run proves little.
- Set a lag window. Compare crawl activity in weeks 1 to 4 with citations in weeks 3 to 8. Index crawling can precede appearance in answers; the exact delay differs by engine and no vendor publishes one, so measure your own.
- Classify each URL into the four cases below.
| Case | Logs | Citations | Likely meaning | Next action |
|---|---|---|---|---|
| A | Search-index or user-fetch hits | Cited | Page is reachable and useful for that prompt | Protect it: keep it fast, current and unblocked |
| B | Hits | Not cited | Crawlable, but another source answers the prompt better, or the page does not state the answer plainly | Rewrite the opening answer, add sourced figures, check which URLs are cited instead |
| C | No hits from you | Cited | The engine may use a cached copy, a third-party index or another site's description of you | Check which domain is cited; fix the source it actually reads |
| D | No hits | Not cited | Probably blocked, unlinked or not discovered | Check robots.txt, firewall rules, sitemap and internal links |
Case B is where most of the work is, and it is a content problem, not an access problem. Case C is the one logs alone can never explain, which is why the citation side matters. For the difference between being cited and being named, read citation vs brand mention.
What server logs cannot prove
- That a crawl influenced an answer. A fetch means a bot read the page. It does not mean the page was used. Only the cited sources in the answer show use.
- That a user saw you. A user-fetch hit shows a retrieval happened for someone's question. It does not show your brand was in the reply the person read.
- Training use. A training-bot visit does not tell you whether, or how, the content affects any model's later answers.
- Identity, from the user agent alone. See the verification section above.
- Anything from cached or third-party copies. If an engine reads a description of you on another site, your logs stay empty.
- Human traffic from AI. Referral clicks show up in web analytics, not as bot hits. They are a separate measurement; see measuring AI influence beyond referrals.
Where Noma fits
Noma by Nurdd is an answer engine optimisation (AEO) platform. It asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, the questions your buyers ask, every day, and shows where your brand is named, where competitors are named instead, and which pages the answers draw on. That is the citation half of the join above, the half server logs cannot supply. See it on your own brand: book a demo or start the 3-day free trial. A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged.
For a no-signup first check of citation status, the free brand audit asks ChatGPT, Perplexity and Gemini ten buyer questions and shows whether your brand is named.
A 30-minute checklist for this week
- Export 30 days of raw logs and filter the eight user agents in the table.
- Verify IPs against each operator's published list and drop or flag the rest.
- Run the AI Crawler Checker and compare its verdict with your logs.
- List status codes returned to each search-index bot. Fix any 403, 429 or 5xx on key pages.
- Check whether any verified bot requested /llms.txt, and validate the file with the llms.txt validator.
- Build the URL table with the four cases and pick three case B pages to rewrite.
- Re-run the same prompts in four weeks and compare. Use the AI crawler tracking guide if you need tooling options for the log side.
If a ChatGPT-driven visit to your site is being refused, this check for blocked ChatGPT agents covers the usual causes.
How we researched this
We reviewed vendor crawler documentation (OpenAI and Perplexity directly, Anthropic through the trade press report that cites its page), a published server-log study, a single-site log write-up and existing Noma guides. We also read other published log-analysis guides to see what they cover, and added the URL-level join, the four-case table and the worked verification example. We did not run our own log study and report no Noma results. Facts, user-agent behaviour and plan terms were checked on 2026-11-16.
Next step
Run the log check this week, then add the citation side so you can see which crawled pages actually get used. Book a demo to see Noma on your own brand, or start the 3-day free trial.
See what AI engines say about your brand
Free · no account · or start a 3-day trial
Questions people ask
How do I analyse AI crawler traffic on my website?
Pull 30 days of raw server or CDN logs, filter for AI user-agent tokens, and verify each client IP against the operator's published ranges. Then group verified hits by job (training, search index, user fetch) and count them per URL per week. Page-tag analytics will miss most of these bots.
Which AI crawlers should I look for in my server logs?
Look for GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot and Claude-User from Anthropic; and PerplexityBot and Perplexity-User from Perplexity. Add Googlebot, Bingbot and CCBot as a separate group. Keep training, search-index and user-fetch agents apart in reports.
Can I trust the user agent in my logs?
No, treat it as a claim. In one published week on four small sites, only 12 of 54 requests claiming to be PerplexityBot came from Perplexity's published ranges, though list lag may explain part of that. Verify against the operator's IP list or reverse DNS before you count or allow a bot.
Does a crawler visit mean an AI engine will cite my page?
No. A fetch shows a bot read the page, not that an answer used it. To see use, track a fixed set of buyer prompts and record which URLs are cited, then compare that list with your crawl data by URL.
Does blocking GPTBot or ClaudeBot hide me from ChatGPT or Claude answers?
OpenAI and Anthropic publish separate bots for training, search and user requests, so blocking the training bot does not by itself block the others. Blocking search bots, such as Claude-SearchBot, may reduce your visibility in search results. Decide on each bot separately.
Do Perplexity-User and ChatGPT-User follow robots.txt?
Perplexity's documentation says Perplexity-User generally ignores robots.txt because a person requested the fetch. Search Engine Journal reports that ChatGPT-User may not be governed by robots.txt in the same way as OpenAI's automated crawlers. Use firewall or application rules if you need to control them.
