An eval suite is a fixed set of cases someone thought to write down in advance. Production traffic is not fixed and was not written down in advance — it's whatever real users actually send, including the exact phrasing, tool-call sequence, and edge case that nobody on the team happened to think of when building the eval. A prompt or model change can pass every eval and still regress in production, because passing the eval only proves it handles the cases the eval covers. The fix isn't a bigger eval set, which is always incomplete by construction — it's a rollout process that limits how much real traffic a new version touches before anyone knows whether it's actually better.
Eval-based regression testing (see catching agent regressions before they ship) answers "does this change break any known case" before deploy. Rollout strategy answers a different question: "now that it's passed every known case, how much live, never-seen-before traffic should touch it before we're confident enough to give it all of it." Both are needed — a change that fails its evals shouldn't reach canary at all, and a change that passes its evals still shouldn't jump straight to 100% of production, because the evals and production traffic are never the same set.
Before a new version handles a single real response, run it in parallel against live traffic in shadow mode: the incoming request goes to both the current version and the candidate, but only the current version's output is actually returned to the user or acted on by downstream tools. The candidate's output is logged and compared, never shipped. This catches a whole class of problem — a version that would have called the wrong tool, hallucinated a fact, or produced dangerous output — with zero user-facing risk, because nothing it produces ever reaches a user or a side-effecting tool call. It's the cheapest stage in the pipeline precisely because it has no blast radius.
Shadow mode never triggers real side effects, so it can't catch problems that only show up when a version's actions actually run — a tool call that succeeds in a dry comparison but behaves differently against a live database, a rate limit that only trips under real concurrent load. Canary closes that gap: route a small percentage of real traffic — 1-5% is typical to start — to the new version and let it actually act, while the rest keeps running the known-good version. The percentage is the entire risk-control knob: a regression on a 2% canary affects 2% of traffic and is cheap to revert, the same regression at 100% is a full incident.
| Stage | Acts on real users | Catches | Blast radius if wrong |
|---|---|---|---|
| Offline eval | No | Known cases regressing | None — never deployed |
| Shadow | No (logged only) | Unknown-case output differences, with zero action risk | None — output discarded |
| Canary (small %) | Yes, bounded slice | Real side-effect and load-dependent failures | Limited to the canary percentage |
| Full rollout | Yes, all traffic | Nothing new — this is the exposure stage | 100%, no more safety margin left |
Without a metric and threshold agreed in advance, a canary produces a wall of transcripts and no decision — every team ends up eyeballing outputs and disagreeing about whether they look fine. Pick a small number of comparable signals before traffic starts flowing: task success rate, tool-call error rate, user correction/retry rate, cost per request, latency. Write down the threshold that means promote and the one that means roll back before looking at a single result — deciding after the fact almost always drifts toward whatever conclusion the person already expected walking in.
A canary is only a safety mechanism if pulling it back is faster than the harm it can do. That means the traffic-routing decision — which percentage goes to which version — has to be a config flip, not a redeploy: a redeploy that takes ten minutes lets a bad canary keep processing traffic for ten more minutes after someone notices. This is the same reversibility argument that governs any agent action with real-world effects (see designing agent actions to be reversible) — applied here to the rollout mechanism itself rather than to what the agent does once it's live.
The test to apply: name the last prompt, model, or tool-schema change that went to production. Did any percentage of real traffic run it before every user's traffic did? If the honest answer is "no, we changed it and it went out to everyone at once," the eval suite was doing all the risk-catching work by itself — and the first regression it misses, by definition the kind it wasn't written to catch, will be found by every single user at the same time instead of by a bounded slice of them.
Related: a canary that regresses is exactly the kind of incident a backoff and circuit-breaker design should contain automatically, and exactly the kind of signal that should turn into a permanent feedback memory entry once the cause is understood, so the same regression shape doesn't ship again next time.