# AI Agent Website Access: Block, Allow or Serve Them

> AI agent website access: decide whether to block, allow or serve each bot, with the robots.txt rules, bot settings and log checks for each choice.

- Author: Sheetal Shakshi, Founder's Office, Nurdd
- Published: 2026-11-09
- Publisher: Noma by Nurdd (https://nomahq.app)
- Original page: https://nomahq.app/blog/ai-agent-website-access

Should you let AI agents crawl your website? Usually yes for search and user-triggered bots, a deliberate choice for training crawlers, and a firm policy for agents that act as your customers. AI agent website access is not one setting: it is a decision per bot type, checked in robots.txt, your bot rules and your server logs.

## Key takeaways

- Split AI traffic into four types before you write any rule: training crawlers, search crawlers, user-triggered fetchers and agents acting for a customer. Each needs a different answer.
- Robots.txt only governs bots that choose to obey it. [ppc.land, March 2026](https://ppc.land/google-agent-joins-the-crawler-list-as-ai-browsing-gets-an-official-identity/) reports that Google's user-triggered fetchers, including Google-Agent, ignore it.
- Blocking GPTBot does not remove you from ChatGPT search. That is a separate bot, OAI-SearchBot, with its own robots.txt token.
- Pick one stance per bot type: block, allow or serve. "Serve" means making the page or the task easy for the agent to complete.
- Check every rule against your logs. A CDN rule can block a bot that robots.txt allows, and you will not see it in robots.txt.
- Use the [free llms.txt generator and validator](https://nomahq.app/tools/llms-txt) and the [free AI crawler checker](https://nomahq.app/tools/ai-crawler-checker) for a first pass today.

## What are the three ways businesses treat AI agents?

Search Engine Journal columnist Slobodan Manic described three answers from large businesses in October 2026, in a piece republished on [No Hacks, November 2026](https://nohacks.co/blog/businesses-treat-your-ai-agent-three-ways). They map onto the choices a marketing or SEO lead faces on their own site.

- **Block.** JPMorgan Chase chief executive Jamie Dimon said on the Q3 2026 earnings call: "we are not gonna allow just any agent to come, outside agent to come inside the company." The same article reports that Charles Schwab blocks agents from accounts and that Delta is preventing them from booking tickets.
- **Allow, at the customer's risk.** Do nothing and the agent signs in with the customer's own login. Bank of America told customers they are responsible for activity that results from sharing a login with an agent.
- **Serve.** Walmart's Agent Profiles give each agent a dedicated profile within the customer's account, with approval per purchase or a spending limit the customer sets, for example $50, $100 or $200.

Those three cases are about agents that transact. Most sites face an earlier question: which AI bots may read the pages at all. The same three words work for both, so the rest of this guide uses them as a decision list. For the wider picture, see [what AEO is](https://nomahq.app/blog/what-is-aeo).

## Which AI bots visit your site, and which rules reach them?

The mistake that causes most accidental blocking is treating "AI bot" as one thing. OpenAI's [crawler documentation, as checked on 8 November 2026](https://developers.openai.com/api/docs/bots), lists separate user agents for separate jobs, including OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, each with its own published IP list. Anthropic splits its bots the same way, according to [Search Engine Journal, February 2026](https://www.searchenginejournal.com/anthropics-claude-bots-make-robots-txt-decisions-more-granular/568253/): ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for fetches a user asks for.

| Traffic type | Examples | Does robots.txt reach it? | Default stance | What to check |
| --- | --- | --- | --- | --- |
| Training crawler | GPTBot, ClaudeBot, Google-Extended | Yes, these are documented as honouring it | A business decision: allow or block | Robots.txt group exists; logs show the bot stops |
| Search crawler | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Yes | Allow if you want to be cited | Not blocked at CDN; pages return 200 |
| User-triggered fetcher | ChatGPT-User, Claude-User, Google-Agent | Varies by vendor: see below | Allow; control with bot rules, not robots.txt | Fetches succeed for public pages |
| Agent acting as a customer | Browser agents using a customer's account | Not designed for it | Block, allow or serve per task | Login, checkout and form paths in logs |

The user-triggered row is where vendors differ. Search Engine Journal reports that OpenAI warns robots.txt rules may not apply to ChatGPT-User and that Perplexity says they generally do not apply to Perplexity-User, while Anthropic says all three of its bots, including Claude-User, honour robots.txt. Google is different again: per ppc.land, it added Google-Agent to its user-triggered fetchers list on 20 March 2026, and that category ignores robots.txt because a person requested the fetch.

OpenAI's crawler overview lists each user agent and its published IP file. Source: OpenAI.

## Should I let AI agents crawl my website?

Work through the stance per type rather than as a yes or no. This is the decision list.

### Stance 1: Allow search and user-triggered bots

If buyers ask ChatGPT, Claude or Perplexity for options in your category, the search and fetch bots are how your pages become candidates for the answer. Blocking them removes a route to being named. Anthropic warns that blocking Claude-SearchBot may reduce visibility in search results, as [Search Engine Journal, February 2026](https://www.searchenginejournal.com/anthropics-claude-bots-make-robots-txt-decisions-more-granular/568253/) reports. Our [guide to ChatGPT search and OAI-SearchBot](https://nomahq.app/blog/chatgpt-search) covers how those citations show up.

What to check: robots.txt has no blanket *Disallow: /* under *User-agent: ** that you forgot about, your CDN is not challenging these bots, and your key pages return a 200 status without a login or script-only rendering.

### Stance 2: Decide on training crawlers separately

Training bots collect content for model training. Blocking them is a legitimate choice and does not, by itself, remove you from search features, because those use different bots. Google also has a separate control: its [common crawlers documentation, as checked on 8 November 2026](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), lists Google-Extended among the tokens you can address in robots.txt. For the wording, see our [guide to Google AI training control in robots.txt](https://nomahq.app/blog/google-ai-training-control-robots-txt) and the [AI Crawler Policy Kit](https://nomahq.app/resources/ai-crawler-policy-kit).

What to check: after you add a Disallow group, watch the logs for a week. A bot that keeps fetching either ignores the rule or uses a different token than you wrote.

### Stance 3: Choose block, allow or serve for agents that act as customers

| Stance | Real example | Fits when | Risk | What to check |
| --- | --- | --- | --- | --- |
| Block | JPMorgan Chase, Charles Schwab, Delta, as reported by No Hacks | Money movement, regulated accounts, scarce inventory | You lose customers who delegate tasks | Bot rules fire on login and checkout; logs show blocks |
| Allow, no policy | Agent signs in as the customer | Never a deliberate choice | Agent and customer are indistinguishable in your data | Look for odd session patterns and speed |
| Serve | Walmart Agent Profiles | Repeat purchases, bookings, simple forms | Needs product and engineering work | Agent traffic is labelled and limits are enforced |

Walmart's own reasoning, quoted in the No Hacks piece, is that purchases by a customer and by an agent otherwise appear together without differentiation. That is a measurement problem as much as a security one. If you cannot tell the two apart, you cannot judge what agents are worth to you.

## How do I stop an AI bot that robots.txt does not reach?

Robots.txt is a request. It works on bots that read it and choose to comply. When the bot is not documented as honouring it, or when it is an agent working inside a customer's session, use rules that sit in front of your site.

1. **Identify the bot by more than its name.** User agent strings can be spoofed, a caution Google repeats in its documentation, per ppc.land. Match the user agent against the vendor's published IP list. OpenAI publishes JSON files for each bot, such as the one for ChatGPT-User.
2. **Prefer cryptographic identity where it exists.** Cloudflare introduced [signed agents](https://blog.cloudflare.com/signed-agents/), built on Web Bot Auth, so sites can allow or disallow agents apart from other verified bots. Cloudflare's [documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) says that as of 1 July 2026 signed agents count as Verified, with a metadata field separating direct from intermediary access. Google noted experimentation with Web Bot Auth under the identity agent.bot.goog, according to ppc.land.
3. **Write the rule at the CDN or firewall.** Allow search bots by verified identity, rate-limit unknown agents on login, checkout and search, and challenge or block on sensitive paths.
4. **Keep sensitive actions behind authentication.** A rule on the public page does nothing once an agent holds a valid session.

Compliance is not guaranteed even when a bot is meant to comply. The publisher-tracking company TollBit found that about 15% of AI page-fetching agents in Europe reached URLs that robots.txt disallowed in the first half of 2026, according to [i-scoop, October 2026](https://www.i-scoop.eu/websites-blocking-ai-agents/). Treat that as one reported figure from one tracker, not a universal rate.

## How do I see which AI agents visit my site?

Your server or CDN logs are the source of truth. Analytics tags miss most bots because they do not run JavaScript, and a robots.txt file tells you what you asked for, not what happened. Here is a one-hour method.

1. **Export seven days of raw access logs** from your host or CDN. Keep the user agent, IP address, path, status code and timestamp.
2. **Search for the documented tokens.** Filter case-insensitively for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Agent and Google-Extended, plus any others you care about.
3. **Verify the IPs.** Compare a sample of requests against each vendor's published IP file. Mark anything that claims a bot name from an unlisted address as suspect.
4. **Group by status code.** A search bot that gets 403 or 429 on your key pages is blocked, whatever robots.txt says. OpenAI's documentation says robots.txt fetches may carry an extra marker in the user agent, so you can separate those requests from page requests.
5. **List the paths each bot type reaches.** Training and search bots should hit articles and product pages. A fetcher on login, cart or account paths is the signal to write an agent rule.
6. **Record the result** in a table with one row per bot: allowed or blocked by policy, observed behaviour, action.

A worked reading, using user agent strings OpenAI documents: a request containing ChatGPT-User/1.0 on one of your pricing pages means a person in ChatGPT asked something that led to that page being opened. A request containing OAI-SearchBot/1.4 is the search index crawler. Same vendor, different intent, and different consequences if you block it.

The [free AI crawler checker](https://nomahq.app/tools/ai-crawler-checker) shows whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google AI can reach your site and generates a corrected robots.txt, which makes it a quick first test before you read logs. For why a site can look open yet still get blocked, see our post on [checking and fixing a blocked ChatGPT agent](https://nomahq.app/blog/chatgpt-agent-blocked-site).

## What does "serve" mean in practice?

Serving an agent means removing friction for the tasks you want it to complete, and labelling the traffic so you can measure it. Cloudflare's data, reported by [TechCrunch, July 2026](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/), suggested over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages. That is a reason to make serving cheap: stable URLs and accurate update dates cut wasted fetches.

- **Make facts quotable.** Put prices, specifications, opening hours and policies in plain HTML text, not only in images or scripts.
- **Keep URLs stable** and return honest last-modified dates so repeat fetches are cheap.
- **Offer a machine-readable map.** An llms.txt file lists your key pages for AI tools. Treat it as a low-cost signal, not a control, because no file format forces an engine to read it. The [free llms.txt generator and validator](https://nomahq.app/tools/llms-txt) builds one from your sitemap and checks every link in an existing file. See also our [llms.txt guide](https://nomahq.app/blog/llms-txt-guide).
- **Use structured data** for products, organisations and FAQs. Our [schema generator](https://nomahq.app/tools/schema-generator) builds the JSON-LD.
- **For transactions, add an agent identity.** Walmart's separate profile with a spending limit is the clearest published design: the agent is visible, linked to the customer and capped.

## How to measure what agents do after you set the rules

A policy you cannot measure drifts. Three things need a number each week: which agents visit, what they fetch, and whether answers that cite you follow. Agent analytics is the name for the first two. It reads server logs and names the agent behind each request, as covered in our explainer on [what agent analytics is](https://nomahq.app/blog/what-is-agent-analytics).

Noma by Nurdd is an answer engine optimisation (AEO) and AI visibility platform. Its [agent analytics](https://nomahq.app/capabilities/agent-analytics) shows which AI agents, such as GPTBot, ClaudeBot and PerplexityBot, read your site, from your server logs, and names the agent behind each request. Alongside that, Noma asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, your buyers' questions daily, so you can see whether crawl access is turning into mentions.

See it on your own brand: [book a demo](https://nomahq.app/demo) or [start the 3-day free trial](https://platform.nomahq.app/signup). A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged.

## Common mistakes with AI agent website access

- **Blocking GPTBot and assuming ChatGPT has gone.** Search and user fetches use other bots, per OpenAI's documentation.
- **Copying a "block all AI" robots.txt snippet** that also names search bots, then wondering why citations fall.
- **Relying on robots.txt for agents.** It does not govern user-triggered fetchers in Google's case, and vendors differ elsewhere.
- **Trusting the user agent alone.** Without IP or signature checks, a spoofed name looks identical to the real bot.
- **Setting no policy for logged-in agents.** You inherit the "allow, at the customer's risk" stance by default.
- **Never rechecking.** Vendors add bots, as Google did on 20 March 2026. Review the list quarterly.

## Checklist: the policy in one page

1. List your bot types and write one stance per type: block, allow or serve.
2. Write robots.txt groups per token, not a single wildcard.
3. Mirror the policy in CDN bot rules, using verified identity where available.
4. Run the free crawler checker, then a seven-day log review.
5. Fix any 403 or 429 on search bots, and any fetches of paths you meant to protect.
6. Publish quotable facts and an llms.txt, and validate them.
7. Set a weekly measure: agents seen, paths reached, mentions and citations gained.

## How we researched this

We reviewed vendor documentation from OpenAI, Google Search Central and Cloudflare, trade coverage from Search Engine Journal, TechCrunch, ppc.land and i-scoop, and the No Hacks article that sparked this topic. Facts, dates and figures were checked on 2026-11-08. Vendor bot lists change often, so confirm user agents on each vendor's page before editing rules. We did not run our own crawler tests for this guide.

## What to do next

Write the one-page policy this week, then verify it in your logs. If you want the logs read for you and tied to whether AI engines name your brand, [book a demo](https://nomahq.app/demo) or [start the free trial](https://platform.nomahq.app/signup).