robots.txt for AI agents: which bots to allow and block

Every AI company publishes its own crawler names, and merchant guides tend to answer one question: how do I block them? For a store that wants to sell to AI shopping agents, that is the wrong question. The useful one is which AI clients to let in, where, and why.

This guide sorts the AI user agents into the three kinds that matter, gives a robots.txt you can copy, and explains where robots.txt stops and your bot manager takes over.

What are the three kinds of AI crawler?#

Search crawlers, user-triggered assistants, and training crawlers. Each major AI company now runs separate user agents for each, so you can treat them separately.

OpenAI's documentation states the separation directly: blocking GPTBot keeps content out of training and does not affect search, which OAI-SearchBot controls. Anthropic splits its agents the same way. Google-Extended is a token rather than a crawler: it has no user agent of its own and only tells Google whether content may be used for Gemini training and grounding. It does not affect Google Search.

Which AI user agents should a store allow?#

Allow search crawlers on product and category pages, and allow user-triggered assistants everywhere a shopper goes, including the basket and checkout. Keep accounts and admin private for everyone.

The reasoning is simple once the kinds are separate:

• Search crawlers decide whether you appear. If OAI-SearchBot cannot read your products, ChatGPT search cannot recommend them. They have no reason to index your basket or checkout, so those can stay disallowed for them.
• Assistants are customers. A ChatGPT-User or Claude-User request is a person's assistant looking at your store on their behalf. Disallowing the purchase path for them tells the agents that honour robots.txt not to buy.
• Training is a separate choice. Whether models may learn from your product copy is about your content, not your sales. Many merchants opt out; it costs them nothing in AI shopping.

What should your robots.txt say?#

Use separate groups for each kind of client, with the paths each needs. Pick your policy below and copy the result. Multiple User-agent lines can share one group, and a crawler follows the group that names it most specifically.

# Search and answer engines: index products, skip private and purchase pages
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /orders
Disallow: /cart
Disallow: /checkout

# Assistants fetching for a shopper: allow the whole purchase path
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /admin
Disallow: /account
Disallow: /orders

# Model training: opted out (does not affect search or shopping)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin
Disallow: /account
Disallow: /orders
Disallow: /cart
Disallow: /checkout

Sitemap: https://your-store.example/sitemap.xml

Two notes on the output. Allow: / followed by Disallow: lines works because the most specific matching rule wins, so /cart stays disallowed for search crawlers while product pages are open. And the final * group covers every client not named above, which keeps unknown bots out of accounts and checkout while leaving the rest of the site as it was.

Does robots.txt actually control AI shopping agents?#

Partly. It is a request that well-behaved crawlers honour, and user-triggered assistants may not follow it at all. It is not access control.

OpenAI says this about ChatGPT-User: because its actions are initiated by a user, robots.txt rules may not apply. Third-party documentation describes Perplexity's user-triggered agent the same way. ChatGPT's browsing agent has no robots.txt token at all; it identifies itself with a Web Bot Auth signature instead. So a robots.txt that disallows checkout will not reliably stop an assistant from reaching it, and one that allows checkout will not get an assistant past a bot manager that blocks it.

That second case is the common one. A store can welcome every AI agent in robots.txt and still refuse them at the basket, because its bot management product applies its own policy. robots.txt says what you want. Your firewall decides what happens. If agents are not completing purchases, look there next: why AI shopping agents get a 403 at your checkout.

What else belongs in robots.txt for agents?#

A sitemap line, and nothing that blocks the pages agents use to answer questions. Three small things make a measurable difference:

1. List your sitemap. Sitemap: https://your-store.example/sitemap.xml tells every crawler where your product URLs are. A sitemap that lists no product pages is nearly as bad as none (sitemap lists no product pages).
2. Do not disallow policy pages. Shipping, returns and cancellation policies answer the questions shoppers ask assistants most. Blocking them makes the assistant guess or say it does not know (no returns path for agents to find).
3. Do not block everything by accident. A single Disallow: / under User-agent: *, left over from a staging site, hides the whole store (robots.txt blocks every agent).

Which AI user agents should you know about?#

About a dozen tokens cover the AI clients that matter to a store in 2026. Names change, so check each operator's documentation before you rely on one.

Two agents you will not find in robots.txt: ChatGPT's browsing agent, which identifies itself with a Web Bot Auth signature instead of a token, and general browser agents that look like ordinary Chrome. For those, the bot manager is the only control there is; see Web Bot Auth explained.

What are the common robots.txt mistakes for AI agents?#

Blocking a whole company because of one bot, and copying a "block all AI" snippet without reading it.

• One rule for every bot from a company. Disallowing GPTBot to opt out of training is fine. Copying the same rule to OAI-SearchBot removes you from ChatGPT search, which was probably not the intent.
• Blocklists from the internet. Snippets that "block all AI crawlers" often include search and assistant tokens alongside training ones. Read each line before you paste it.
• Wildcards that catch too much. Disallow: /*? to stop crawlers on filtered URLs can also block product URLs that carry a variant parameter.
• Different files on different hosts. robots.txt applies per host. shop.example and www.shop.example each need the right file, and so does any checkout subdomain.

How do you check it is working?#

Test it per user agent, then test what the server does. robots.txt testers show what a rule means; only a request shows what your stack does.

• Parse it for each token. Google Search Console's robots.txt report shows how Google reads your file. For other tokens, a robots.txt parser library or a scanner that evaluates rules per user agent tells you which paths each one may fetch.
• Request the paths. Fetch a product page, the basket and the first checkout step as each kind of agent and read the status codes. A path robots.txt allows but your firewall refuses is the failure that costs sales.
• Re-check after platform updates. Theme changes, apps and platform upgrades can rewrite robots.txt. A check in your deploy pipeline catches it; see how to test your store with AI shopping agents in CI.

Written by the tragent team and checked against the sources listed above on October 19, 2026. See how we write and correct posts in our editorial policy.