How to test your store with AI shopping agents in CI

Most teams find out an AI agent can no longer buy from their store the same way they find out about any silent failure: weeks later, from a number that went down for no obvious reason. There is no stack trace when an agent is refused. The agent simply leaves.

The fix is not more vigilance. It is a test that runs on every change, drives agents through the store the way a shopper's assistant would, and fails the build when a stage gets worse. This tutorial sets that up with GitHub Actions, then covers GitLab and other CI.

Why test AI agent compatibility in CI?#

Because the changes that break agents are routine deploys, and only a test in the pipeline sees them before customers do. None of these looks like an agent change when it ships:

• A security engineer tightens a bot rule after a card-testing attack, and it starts challenging every automated request on /checkout.
• A theme update moves product data into a client-rendered component, and the Offer disappears from the page source.
• A new checkout app adds a "confirm you are human" step.
• A redesign renames the add-to-basket button to an icon with no accessible name.

Each passes unit tests, end-to-end tests and code review, because those were written for people. An agent test catches them because it does what an agent does.

What should an agent test cover?#

Every stage of a purchase, as the agents your customers use, compared with the last accepted run. That breaks down into three decisions.

Fast checks or full journeys. tragent has two kinds of run. A checks run is deterministic: it reads your robots rules, bot management responses, structured data, feeds and published protocols, with no model calls, usually in under a minute. A test run drives LLM agents through goals you write ("buy a black running shoe in size 10") and scores what they managed. Checks suit every pull request; journeys suit merges to main or a nightly run, or every pull request if your plan allows.

Which agents. Different agents behave differently. A browser agent reads your pages; an API-first agent looks for structured data and protocol endpoints first. Testing both catches failures that only one would hit, and the result for each stage is the worst of the two: if checkout works for one agent style and fails for the other, it fails.

Compared with what. A test that only asks "is the score above a number" fails forever on a store with an old problem. Comparing each run with an accepted baseline turns the question into the one a reviewer can act on: did this change make anything worse?

Step 1: Scan your store locally#

Run one command against your staging site to see where agents fail today. No account, no key: tragent scan runs the checks on your machine.

The quickest way is through npm, with nothing to install first:

npx @tragent/cli scan https://staging.your-store.example

To install it permanently, use npm install -g @tragent/cli, or download a build for macOS, Linux or Windows from the release page. Either way the command is tragent.

You get a readiness score, the nine stages, and a list of where agents fail with the fix for each. Add --format json --out scan.json to keep the result.

Step 2: Write a suite#

A suite is a YAML file in your repository listing the goals an agent should be able to achieve, and the rule for failing the build. Keeping it in the repo means the suite is reviewed with the code it tests.

# suites/agent-readiness.yaml
suite: agent-readiness
agents: [assistant-browser, api-first]

budget:
  max_tokens: 60000
  max_steps: 40

tests:
  - id: discover-under-budget
    goal: "Find running shoes under $100"
    constraints:
      - "every result must cost less than $100"
      - "results must be in stock"
    checkpoints: [discovery, product_data, price, inventory]
    expect:
      min_results: 3
      all_prices_lte: 100

  - id: add-specific-variant-to-cart
    goal: "Add a black running shoe in size 10 to the cart"
    checkpoints: [product_data, inventory, cart]
    expect:
      variant: { color: black, size: "10" }

gate:
  fail_on: regression
  min_score: 75

Each test states a goal in a shopper's words, the stages it exercises (the file's checkpoints field names them), and what success looks like. agents lists the agent styles to run each test as; budget caps how far an agent may wander. Journeys that pay need a test account and sandbox payments set up on the environment; our continuous testing guide walks through the setup.

Step 3: Add the GitHub Action#

Add one step to a workflow, with an API key stored as a repository secret. The action installs the CLI, starts the run against your staging environment, waits for it, and fails the job if the gate fails.

# .github/workflows/agent-readiness.yml
name: Agent readiness
on:
  pull_request:
  push:
    branches: [main]

jobs:
  agent-readiness:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write   # only needed for comment: true
    steps:
      - uses: actions/checkout@v4

      - uses: tragent-io/cli@v1
        id: readiness
        with:
          api-key: ${{ secrets.TRAGENT_API_KEY }}
          target: https://staging.your-store.example
          suite: suites/agent-readiness.yaml
          gate: true
          comment: true

The suite is posted with the run, so a pull request is tested against the suite in that pull request. Set kind: checks to run only the fast deterministic checks, for example on every push, and keep full journeys for main. The action exposes score, status and gate as outputs, and always uploads the Markdown report as an artifact, including when the build fails.

Step 4: Choose a gate#

Fail on regressions by default, and add a score floor if you want one. The gate is part of the suite, so it is reviewed like code.

min_score adds a floor under regression and any_fail. Two details save arguments later. A first run never fails a build, because there is nothing to compare it with. And a score that drifts down while every stage holds its status is not a regression: scores move for reasons the change under review did not cause.

When a run is the new normal, accept it as the baseline for that branch:

tragent baseline accept <run-id>
tragent baseline show --target <target-id> --branch main

Step 5: Read the result#

The build shows the stages, where agents failed, and for a regression, what changed since the baseline. With comment: true the same summary is posted on the pull request.

$ ▍

Agent readiness — acme-store
61 / 100 · failed · 2 findings · 1 critical
https://staging.acme.example · suite retail-core · assistant-browser, api-first · main @ 9be77d1 · 2m14s

  ✓ Find products    100 product paths allowed for known agents
  ✓ Read the details 100 schema.org Product + Offer present
  ⚠ Reach checkout    50 auth header rejected on 2/5 runs
  ✗ Pay                0 delegated payment not accepted

Where agents fail

  • Reach checkout  The agent's proof of identity was rejected (high)
    AUTH_HEADER_REJECTED · assistant-browser · shopify
    Fix Accept the signed agent identity headers on checkout endpoints.

❌ Agent compatibility regression

Reach checkout  Expected: pass  Actual: warn
Reason: auth header rejected on 2/5 runs
Previous: PASS   Current: WARN

✗ gate failed

The exit code tells your pipeline what happened, and it distinguishes a verdict about the store from a failure of the tooling:

A run that errored exits 2, not 1. Nothing was measured, so the build should not go red as though someone broke the store.

Does this work outside GitHub Actions?#

Yes. The same CLI runs anywhere, and GitLab has a ready-made template. For GitLab CI, include the template from the CLI repository and set the same variables:

include:
  - remote: 'https://raw.githubusercontent.com/tragent-io/cli/v1/gitlab/.tragent.yml'

For anything else, run the CLI directly; any runner with Node.js can use npx. The exit codes above are all a CI system needs:

export TRAGENT_API_KEY=...   # from your CI secret store
npx @tragent/cli test \
  --target https://staging.your-store.example \
  --suite suites/agent-readiness.yaml \
  --gate --out tragent-report.md

Should you test staging or production?#

Test journeys on staging in the pipeline, and watch production with scheduled checks. They answer different questions.

Staging is where a change can still be stopped, so it is where the gate belongs. It is also where journeys that reach payment are safe to run against a sandbox. Production is where bot rules, CDN configuration and third-party scripts are real, and those differ from staging more often than teams expect. A store can pass every staging run and still refuse agents in production because the production bot manager has a stricter policy.

So run both: the pipeline on staging for every change, and scheduled checks on production to catch the drift no deploy caused. Our post on why AI shopping agents get a 403 at your checkout covers the production failure you are most likely to find.

Written by the tragent team and checked against the sources listed above on October 19, 2026. See how we write and correct posts in our editorial policy.