acurl: see your site the way AI agents see it

Ask most store owners whether ChatGPT can read their product pages and the honest answer is "I think so". The page loads in a browser. The bot management dashboard says AI crawlers are allowed. But nobody has looked at what comes back when the request says ChatGPT-User instead of Chrome, and that is where the surprises are: a challenge page, a thinner page built for crawlers, or a cached price from another region.

We built acurl to make that check take one command. It is free, open source under the Apache 2.0 licence, and it runs on your machine: nothing is sent anywhere except to the site you are checking. tragent sells agent testing, so we have an interest in this subject; acurl is the quick, free first check, and this post is written so its advice holds whether or not you ever use tragent.

What does acurl do?#

acurl requests the same URL once per agent profile, compares each response with what a real Chrome visit received, and gives every agent one verdict. A profile is a user agent string, the headers that agent sends, and the robots.txt token it obeys.

The browser is the baseline. acurl fetches it with Chrome's TLS and HTTP/2 fingerprint and Chrome's client hints, so the site sees what looks like an ordinary visit. Bot profiles send only their own user agent and headers over a standard TLS stack, the way most real crawlers do. If you want the user agent to be the only difference, --tls chrome gives the bots Chrome's fingerprint too.

Each agent then gets exactly one verdict:

The verdicts are checked in that order of severity, so an agent that gets a Cloudflare challenge with a 403 status is reported as challenged, not merely blocked. That distinction matters for the fix: a challenge comes from bot management, a plain 403 more often from a firewall rule or the application itself.

What does a run look like?#

One table, one score, and a list of fixes. To show every kind of failure on one screen, we pointed acurl at a demo store running on our own machine. It challenges ChatGPT-User the way Cloudflare does, serves OAI-SearchBot a nearly empty page, quotes Perplexity-User a lower price, and disallows GPTBot in robots.txt. This is the real output, trimmed to the rows that matter:

$ acurl --category ai-assistant,ai-search,ai-training http://127.0.0.1:8099/products/trail-runner
Fetching http://127.0.0.1:8099/products/trail-runner with 19 profiles...

PROFILE            STATUS  RESULT             DETAILS
Chrome (Browser)   200     allowed            Trail Runner | Demo Store
ChatGPT-User       403     challenged         challenge: Cloudflare
OAI-SearchBot      200     cloaked            0% similar to browser
GPTBot             200     robots-disallowed  robots.txt: disallowed
Claude-User        200     allowed            OK
Perplexity-User    200     allowed            product: price 89.00 vs browser 98.00
Google-Extended    -       allowed            robots.txt token (not fetched)

AI readiness 88/100 (B)
  ✗ AI assistants can fetch the page               20.0/25  4 of 5 allowed: ChatGPT-User (challenged)
  ✗ AI search crawlers can fetch the page          15.0/20  3 of 4 allowed: OAI-SearchBot (cloaked)
  ✗ No bot challenges for AI agents                 9.4/10  challenged: ChatGPT-User
  ✗ AI agents see the same content as browsers      9.4/10  different content for: OAI-SearchBot
  ✓ Content is in the HTML without JavaScript      15.0/15  page text is in the raw HTML
  ✓ Product structured data is present             10.0/10  JSON-LD Product with price and availability
  ✗ AI agents see the same price and availability   9.4/10  different product data for: Perplexity-User

Four things stand out, and each needs a different person to fix it.

• ChatGPT-User is challenged. An assistant fetching a page because a shopper asked about it cannot solve a JavaScript challenge, so for that shopper the product does not exist. This is a bot management setting.
• OAI-SearchBot is cloaked. It got a 200, but the page has none of the product's text. A status-code check would have passed it.
• Perplexity-User was quoted a different price. acurl reads the schema.org Product data from each response and compares it with the browser's. Caching keyed on the user agent, or geo-pricing applied to a datacenter IP, are the usual causes.
• GPTBot is disallowed in robots.txt. That may be exactly what you want. Blocking a training crawler costs nothing in the score; the score only counts training crawlers when they are challenged or cloaked.

Below the table, every failed check comes with a plain-language fix, so the output can be pasted straight into a ticket. Our post on why AI shopping agents get a 403 at your checkout goes deeper on the bot management settings behind the first finding.

How do you see what an agent missed?#

--diff prints the visible text the browser got next to the text the agent got. acurl compares visible text, not raw HTML, so nonces, CSRF tokens and inline scripts do not count as differences. For the cloaked result above:

$ acurl --agents "Chrome (Browser),OAI-SearchBot" --diff OAI-SearchBot http://127.0.0.1:8099/products/trail-runner
--- Chrome (Browser)
+++ OAI-SearchBot
- Trail Runner | Demo Store
- Trail Runner
- A lightweight trail shoe with a grippy outsole, a breathable mesh upper and a cushioned midsole for long days on rough ground.
- Free shipping on orders over $50. Free returns within 30 days. Sizes 6 to 14, in four colours.
- Weight: 280 g
- Drop: 6 mm
- Upper: recycled polyester mesh
- Rated 4.7 out of 5 by 1,180 runners, who mention the grip on wet rock and the roomy toe box most.
+ Demo Store
+ Welcome to Demo Store.

The same comparison drives the cloaked verdict: below 50% similarity by default, adjustable with --cloak-threshold. With -f html -o report.html, the diffs go into a self-contained report you can attach to a ticket or share with an agency.

Two related problems hide in the raw HTML rather than the response status. If the page is a client-rendered shell, the text only appears after JavaScript runs, and most agents never run it. acurl flags those shells, and --render loads the page in a local headless Chrome to measure how much of the text depends on JavaScript. Our post on product structured data for AI agents covers the second: making price and stock readable without rendering at all.

What about robots.txt and llms.txt?#

acurl robots reads only robots.txt and llms.txt, and gives a verdict per agent without fetching the page. It matches rules with Google's open-source robots.txt parser and follows RFC 9309 for failures: a 404 robots.txt allows everything, while a 5xx or an unreachable server means crawlers must assume they are blocked.

$ acurl robots --agents GPTBot,ChatGPT-User,ClaudeBot,Google-Extended http://127.0.0.1:8099/products/trail-runner
robots.txt audit for http://127.0.0.1:8099/products/trail-runner
http://127.0.0.1:8099/robots.txt (200)

AGENT            TOKEN            CATEGORY      ALLOWED  CONTENT SIGNALS
ChatGPT-User     ChatGPT-User     ai-assistant  yes      -
GPTBot           GPTBot           ai-training   no       -
ClaudeBot        ClaudeBot        ai-training   yes      -
Google-Extended  Google-Extended  ai-training   yes      -

llms.txt: not found (404)

llms-full.txt: found, 0 links in 0 sections
  ! served HTML instead of Markdown (probably a catch-all page)

The last line is a real finding from our demo store, and a common one on live sites: the server answers every unknown path with a normal page, so /llms-full.txt returns HTML with a 200. A tool that only checks the status code would report the file as present. The audit also reads Content-Signal directives such as ai-train=no, and the main command reads X-Robots-Tag and meta robots tags, including agent-specific ones like <meta name="GPTBot" content="noindex">. Our guide to robots.txt for AI agents explains which tokens to allow and which you can safely block, and does llms.txt work? covers whether the file is worth publishing at all.

How do you run it in CI?#

Treat agent access like any other regression: check it on every pull request and fail the build when it gets worse. The exit code follows a --fail-on policy. The default fails on blocked, challenged and network errors; you can add cloaked, robots, rate-limited or a score threshold:

acurl --category ai-assistant,ai-search \
      --fail-on 'blocked,challenged,cloaked,score<80' \
      https://staging.shop.example/products/best-seller

For more than one page, acurl batch takes a list of URLs, a CSV file or a sitemap, spaces out requests to each host, and writes one report for the whole run. -f sarif turns failures into GitHub code scanning alerts, and -f junit works with the test reporters in GitLab, Jenkins and most other CI systems. There is also a GitHub Action that installs acurl, writes a job summary and uploads the SARIF file:

- uses: tragent-io/acurl@v1
  with:
    urls: |
      https://staging.shop.example/
      https://staging.shop.example/products/best-seller
    fail-on: blocked,challenged,score<80

For changes no deploy caused, such as a bot management vendor changing a default, acurl watch re-checks pages on a schedule and posts to Slack or a webhook when any agent's verdict changes. For scripts, -f json exposes everything, so a shell one-liner can list the agents that need attention:

acurl -q -f json --category ai-assistant https://shop.example \
  | jq -r '.results[] | select(.classification != "allowed") | "\(.profile)\t\(.classification)"'

What acurl cannot tell you#

acurl shows how a site responds to agent user agents from your network. It does not prove what a real agent will experience, and it does not test buying. Three limits are worth knowing before you rely on a green result.

• It sends requests from your IP address, not the agents'. Real AI agents come from published IP ranges, and bot managers increasingly verify them that way, or with cryptographic signatures. A site that trusts ChatGPT's published IP ranges may challenge acurl's spoofed ChatGPT-User and still let the real agent through, and the reverse is possible too. Run acurl from the network your users would see; results from a cloud CI runner can differ from your laptop, because many firewalls score datacenter IPs. If the browser baseline itself is challenged, acurl does not score the page at all, because the site is blocking your network rather than the agents.
• Signatures are partial. acurl can sign its bot requests with HTTP Message Signatures (RFC 9421) and an Ed25519 key, which is useful for testing your own verifier. Its signing is checked against the RFC's test vector, but Web Bot Auth is still an Internet-Draft and its header format has moved on, so confirm acurl's signatures against your verifier before trusting the result.
• Reading a page is not buying from it. acurl answers whether an agent can read the product page. It does not add to basket, reach checkout or pay. A store can pass every acurl check and still lose every agent at a CAPTCHA on the payment step.

That last gap is the one tragent exists for: its agents walk the nine stages of a purchase, from finding products to paying and returning an order, on every deploy. If you want that in CI, our tutorial on testing your store with AI shopping agents in CI shows how. For many stores, acurl is the right place to start: free, a minute to install, and enough to find the most common failure, an assistant turned away before it ever sees the page.

How to install acurl#

acurl is a single binary for macOS, Linux and Windows. Pick whichever installer you already use:

brew install tragent-io/tap/acurl                          # Homebrew
curl -fsSL https://acurl.dev/install.sh | sh                # checks the SHA-256 checksum
npx @tragent/acurl https://yourstore.com                    # npm, nothing to install
go install github.com/tragent-io/acurl/cmd/acurl@latest     # Go 1.26 or newer
docker run --rm ghcr.io/tragent-io/acurl https://yourstore.com

Then point it at the page that matters most, usually your best-selling product:

acurl https://yourstore.com/products/best-seller

The documentation on acurl.dev covers every command and flag. The source is on GitHub; bug reports, new agent profiles and detection signatures for other bot managers are welcome.

Frequently asked questions

Is acurl free?

Yes. acurl is open source under the Apache 2.0 licence, with no account, key or usage limit. It runs on your machine and sends requests only to the site you check.

How is acurl different from curl -A?

curl -A sends one request with one user agent and shows you the response. acurl sends one per agent, compares each with a real browser visit, recognises challenge pages, measures how different the visible text and product data are, checks robots.txt for each agent, and turns the result into verdicts, a score and exit codes.

Does a clean acurl result mean ChatGPT can shop on my store?

No. It means the site served the same page to a request carrying ChatGPT's user agent from your network. Real agents use published IP ranges and may sign requests, and buying involves basket, checkout and payment steps acurl does not test.

Will running acurl get my IP blocked?

It makes about twenty requests per page, one per agent, a few at a time, which most sites treat like a normal visitor. acurl batch spaces out requests to the same host by default. If you check a site you do not run, keep runs small.

Can I add my own agent profiles?

Yes. Pass a YAML file with --profiles, listing a name, user agent, optional headers and the robots.txt token for each agent. acurl profiles list shows the built-in catalog.

Does acurl work with Claude or other AI assistants?

Yes. acurl mcp runs a Model Context Protocol server, so Claude, Cursor and other MCP clients can run checks as tools. Private and internal addresses are blocked there by default.

Sources

1. acurl documentation — acurl.dev
2. acurl source code and releases — tragent (GitHub)
3. RFC 9309: Robots Exclusion Protocol — IETF
4. RFC 9421: HTTP Message Signatures — IETF
5. Google's robots.txt parser and matcher library — Google (GitHub)
6. Overview of OpenAI crawlers — OpenAI
7. The /llms.txt file — llmstxt.org

Written by the tragent team and checked against the sources listed above on October 19, 2026. See how we write and correct posts in our editorial policy.