Agent analytics measures how AI crawlers and AI agents visit and use your site. It names the agent behind each request, such as OpenAI’s GPTBot or Anthropic’s ClaudeBot, shows which pages it fetched and how often, and separates bots collecting pages to learn your site from bots fetching a page to answer a question someone has just asked.
It matters because an AI engine can only cite what it can read. If the pages you wrote to be quoted are never fetched, no amount of rewriting will get them into an answer. Agent analytics is how you find that out.
How agent analytics works
AI crawlers do not run JavaScript, so a tracking script on the page never sees them. What does see them is your server: every request leaves a line in its logs, with the path that was asked for and a user agent string that names the bot. Agent analytics reads those lines.
- Collect the logs. Send your server or hosting logs to the tool. Nothing is added to your pages.
- Match each request to an agent. Each AI operator publishes the user agent tokens its bots send, such as
GPTBotorPerplexityBot. A request carrying one is attributed to that named bot rather than counted as unknown traffic. - Sort by purpose. A training crawler, a search crawler and a fetch made for a live question mean different things, so they are recorded as different events.
- Report by page. The result is a list of which agents reach which pages, how often, and which important pages no agent reaches at all.
Agent analytics vs web analytics
Web analytics and agent analytics answer different questions, and most sites need both.
| Aspect | Web analytics | Agent analytics |
|---|---|---|
| Who it counts | People visiting in a browser | AI crawlers and AI agents fetching your pages |
| How it collects data | A script that runs in the visitor’s browser | Your server logs, which record every request |
| What it identifies | Sessions, sources, pages viewed and conversions | The named bot, its operator, its purpose and the pages it fetched |
| The question it answers | What did people do on the site? | Which AI systems are reading the site, and which pages? |
The two meet where an AI assistant sends a person back to you: that visit is a person in a browser, so web analytics can count it, while the fetch that led to the answer only shows up in your logs.
Which AI agents to track (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot)
Start with these four. Each row comes from the operator’s own documentation, linked from the bot’s name and re-read on 20 October 2026; it is the same list Noma’s free AI Crawler Checker uses.
| Bot | Operator | Kind | What the operator says it does | Blocking it can keep you out of AI answers | robots.txt |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training: collects pages to train future models | Crawls content that may be used to train OpenAI's foundation models. | No | Obeys robots.txt |
| OAI-SearchBot | OpenAI | Search: builds the index an assistant searches when it answers | Surfaces websites in ChatGPT's search features. | Yes | Obeys robots.txt |
| ClaudeBot | Anthropic | Training: collects pages to train future models | Collects web content that could help train Anthropic's models. | No | Obeys robots.txt |
| PerplexityBot | Perplexity | Search: builds the index an assistant searches when it answers | Surfaces and links websites in Perplexity search results. Not used to train foundation models. | Yes | Obeys robots.txt |
The difference that matters is the kind of bot. Blocking a training crawler such as GPTBot or ClaudeBot keeps your pages out of future training data, which many publishers choose on purpose; it does not remove you from AI answers. Blocking a search crawler such as OAI-SearchBot or PerplexityBot does, because those are the bots that put your pages in front of the assistant when it answers.
Next, track the fetchers that act for a person in the moment. These are the closest thing to a live answer that you can see in your own logs.
| Bot | Operator | Kind | What the operator says it does | Blocking it can keep you out of AI answers | robots.txt |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | User fetch: reads a page now because someone asked | Visits a page when a user asks ChatGPT or a Custom GPT, or through GPT Actions. | Yes | May ignore robots.txt |
| Claude-SearchBot | Anthropic | Search: builds the index an assistant searches when it answers | Navigates the web to improve the quality of search results for Claude users. | Yes | Obeys robots.txt |
| Claude-User | Anthropic | User fetch: reads a page now because someone asked | Fetches a page when a Claude user asks a question that needs it. | Yes | Obeys robots.txt |
| Perplexity-User | Perplexity | User fetch: reads a page now because someone asked | Visits a page when a user asks Perplexity a question, and may link it in the answer. | Yes | May ignore robots.txt |
Operators change their bots, so re-read their pages before you change a robots.txt rule. For a list of every bot and whether your site lets it in, run the AI Crawler Checker.
What agent analytics shows in Noma by Nurdd
Noma by Nurdd includes agent analytics from the Starter plan. It reads your server logs, not a script on the page.
- Three ways in: a generic webhook, a Vercel log drain, or an access log you upload.
- Named agents: each request is matched on the user agent tokens the operator publishes, so it is attributed to a named bot.
- Crawlers and citations told apart: a model reading your site to learn it and a model fetching a page to answer a live question are recorded as different events.
- Pages: which pages each AI crawler fetches, which AI assistants send visitors back, and the pages that are crawled but never cited.
The agent analytics page describes the feature, and the help article How does agent analytics work? covers setup. For the wider picture of how engines pick sources, read how AI engines choose citations, and for the term next to its neighbours, see the glossary.
FAQ
What is agent analytics?
Agent analytics measures how AI crawlers and AI agents visit and use your site: which agents request your pages, which pages they fetch, how often, and whether they are collecting pages for training, building a search index, or fetching a page to answer a question someone just asked.
Why can't Google Analytics see AI crawlers?
Web analytics tools count visits through a script that runs in the visitor's browser. AI crawlers fetch the page without running that script, so they never appear. Your server sees every request, which is why agent analytics reads server logs instead of a page tag.
Should I block GPTBot?
That is a choice about training, not about AI answers. OpenAI says GPTBot crawls content that may be used to train its models, while OAI-SearchBot surfaces sites in ChatGPT search. Blocking GPTBot keeps your pages out of training; blocking OAI-SearchBot keeps them out of ChatGPT search answers.
What is the difference between a crawler and a citation fetch?
A crawler visit is a model or search index reading your site to learn it. A citation fetch is an agent retrieving your page because someone asked a question and the answer is being written now. Agent analytics records them as different events, because they mean different things for your visibility.
How do I start with agent analytics?
Check first that the bots you want are allowed in your robots.txt, which a free crawler checker can do. Then send your server logs to an agent analytics tool. In Noma by Nurdd that is a webhook, a Vercel log drain or an access log upload, with nothing to add to your pages.
See what AI engines say about your brand
Free · no account · or start a 3-day trial