Issue reference

Your rules block a known shopping agent

Issue code: BOT_RULE_BLOCKS_AGENT
Stage: Find products
Protocol: —
Severity: High

What it means

tragent reports this at the “Find products” stage. The question there is “Can an agent find your products at all?” An agent has to reach your site and locate products by searching or browsing before anything else can happen.

What it does to an agent

Most agents fail at this point. The few that find a way round it spend more time and model calls doing so, and may give up before checkout.

Some agents fetch your pages because a person asked them to, in that moment: ChatGPT-User, Claude-User and Perplexity-User are the user-directed fetchers of the three largest assistants. Others crawl ahead of time to build an index. tragent reads your robots.txt and reports when a known shopping agent is disallowed from the paths a purchase needs: products, search and the basket.

A rule written to keep training crawlers out often catches the user-directed agents as well, because it is written as a wide disallow. The result is that a shopper asks an assistant about your product and it tells them it cannot read your site.

How to fix it

Allow the named agent on product, search and cart paths, or your listings never reach its users.

Before: robots.txt that blocks shopping agents

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-User
Disallow: /

After: Training crawlers out, shoppers' agents in

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /products/
Allow: /collections/
Allow: /search
Disallow: /account

On your platform

Shopify

Edit robots.txt by adding a robots.txt.liquid template to your theme (Online Store › Themes › Edit code › Add a new template › robots.txt). Keep Shopify's default groups and add a group for the user-directed agents after them.

templates/robots.txt.liquid

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules %}
    {{ rule }}
  {%- endfor %}
  {%- if group.sitemap != blank %}
    {{ group.sitemap }}
  {%- endif %}
{% endfor %}

User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /products/
Allow: /collections/

WooCommerce

WordPress serves a virtual robots.txt unless a physical file exists. SEO plugins edit it under their tools menu. Look for groups naming AI agents, which some plugins add when you tick an "block AI crawlers" option, and separate the user-directed agents from the training crawlers.

Cloudflare

Cloudflare can manage robots.txt for you and block AI crawlers at the edge. If either is on, read what it publishes for your zone and check the WAF and bot rules, because an edge block overrides any Allow line in robots.txt.

In practice

Decide the policy before you edit the file. Most stores want three things at once: keep model-training crawlers out, let search and answer-engine indexers in, and always let a fetcher acting for a shopper read product, collection and search pages. Those are different user agents, and robots.txt rules are per agent, so write a group for each rather than one wide rule.

Remember that a group for a named agent replaces the * group for that agent; it does not add to it. If your * group disallows /checkout and /account, repeat those lines in the agent groups you add. And robots.txt is advice, not enforcement: if a bot-management or WAF rule challenges the same agents, the Allow lines change nothing, so check the edge in the same pass.

Check the fix

Open /robots.txt and find every group that names an AI agent.
Check that ChatGPT-User, Claude-User and Perplexity-User are allowed on product, collection and search paths.
Run the scan; the finding should be gone or list fewer agents.

Run the check that reported BOT_RULE_BLOCKS_AGENT again after the change. The scan needs no account; an issue found during an AI shopping test needs that test run again.

Terminal

$ npx @tragent/cli scan https://your-store.com

Common mistakes

Blocking with User-agent: * and a wide Disallow, then adding agent groups that only Disallow more.
Fixing robots.txt while a bot-management rule at the edge still challenges the same agents.
Allowing /products/ but not the search path agents use to find the product in the first place.