Release checks in CI

The deploy that breaks agents should not reach production.

Run the suite on every pull request. Fail the build when a stage moves backwards — and only then.

Exit codes: 0 pass · 1 gate · 2 error
Gate modes: regression · any_fail · never
Integrations: GitHub · GitLab · any CI
First run: never fails

The need

Agent compatibility breaks the way accessibility breaks: quietly, in a change nobody connected to it.

A WAF rule tightens. A checkout step gains a challenge. A framework upgrade changes how a price renders. None of it looks like a commerce change, none of it fails your tests, and the first signal is a number moving in a dashboard weeks later.

The fix is the same one that worked for tests and types: measure it on every pull request, compare it to what was accepted before, and refuse to merge a regression.

The change that breaks agents is usually infrastructure, not commerce code.
By the time a conversion chart shows it, the deploy that caused it is twenty deploys back.
Nobody files a ticket, because nobody experienced it — the customer's agent did.

A nightly synthetic check tells you something broke since yesterday. A pull-request gate tells you which change broke it, while the author is still looking at it.

How it works

One CLI, one gate rule, three exit codes.

Run the suite
tragent test --target … --suite … --gate, from the GitHub Action, the GitLab template, or any CI that can run a binary. Progress streams while it works.
Compare against the reference result
A reference result is a finished test a human accepted, per store, test plan and branch. The release check compares this test against it — not against the last run, which may itself have been broken.
Decide
fail_on: regression fails only when a stage moves backwards. any_fail fails on any failing stage. never reports without blocking. min_score is a floor under all three.
Report back
The Action uploads the report and updates a single pull-request comment in place. An optional signed webhook writes a GitHub check.

whenA run finishes onmain
ifReadiness is under75or any stageregresses
thenBlock the deploy
andComment on the pull request

One comment, edited on every push. A thread of forty bot comments is how a useful signal becomes noise people mute.

What it brings

A gate people trust enough to leave switched on.

A score drop is not a regression
Only a stage moving backwards fails the build. Scores drift for reasons that are not this commit's fault, and a gate that cries wolf gets disabled within a week.
A first run never fails
With no accepted reference result there is nothing to regress against, so the run passes and says why. Otherwise every repository's first green pipeline would be red.
Broken tooling is not a verdict
A test error exits 2, not 1. Nothing was measured, so it is reported as a tooling failure rather than a judgement about your site.
Reference results move when a human says so
Accepting a reference result is an explicit action on a finished, scored test. Nothing promotes itself quietly into the thing you are measured against.

Boundaries

What it does not do.

Stated here so you do not have to discover it in a pilot.

It does not tell you which commit caused a drift.
The gate answers whether this run regressed against the accepted reference result. Attributing a slow decline across many merges is a monitoring question, not a gate one.
It needs a store to test.
A preview URL or a staging store has to be reachable from the runner. tragent cannot gate a deploy it cannot reach.
The webhook is optional and stays off until configured.
It verifies a signature and refuses to guess which tenant a repository belongs to.

See it against your own site. A scan takes about ninety seconds and needs no account.