Eval-Driven Release Gates
Quarterly release reviews gave way to automated evaluation gates that run before every model and agent update ships — cutting release time 71% while incidents fell.
The challenge
The claims operation's triage and adjudication-assist agents were reviewed for release once a quarter, by a board that ran manual test scripts against whatever build was ready at the time. That cadence created its own risk:
- An average of 10.5 weeks between a model update being ready and it reaching production
- Manual test scripts that covered known cases well but missed edge cases and adversarial prompts
- Updates batched together for a single quarterly review, making it hard to isolate which change caused an issue
- Business-rule regressions (a claim type mis-routed, a payout rule dropped) caught by claimants, not by testing
- No way to ship an urgent fix outside the quarterly window without an exception process
The operation logged nine production incidents in the quarter before the change, several traced back to updates that had been batched together and reviewed as a group.
How it works
A gate the update has to clear, not a board it has to wait for
Release cadence didn't change by moving faster through the old process — it changed by replacing the process:
- 01
A golden test set built from two years of adjudicated claims, covering the full range of claim types and edge cases
- 02
An adversarial suite added to probe prompt-injection and edge-case business-rule failures specific to claims agents
- 03
Business-rule checks encoded as automated assertions (payout rules, routing rules, required disclosures)
- 04
A regression suite run against every candidate update before it can be promoted, not just the quarterly batch
- 05
Pass/fail thresholds set per suite, with any failure blocking promotion automatically
- 06
A shadow-traffic stage where passing updates run against live traffic without serving decisions, for a final check
- 07
Release authority delegated to the gate itself — an update ships the day it clears, not the next scheduled review
What we built
Key capabilities
Continuous, not quarterly
Every update runs the full eval suite before it can ship, independent of the calendar.
Adversarial coverage
The suite includes adversarial and edge-case prompts specific to claims workflows, not just known-good test cases.
Business rules as assertions
Payout, routing, and disclosure rules are encoded as automated checks a build must pass, not reviewed by eye.
Shadow traffic before serving
Updates that pass the suite still run against live traffic silently before they're allowed to serve real decisions.
Before vs after
What changed in the release process
- Time-to-release
- 10.5 wks → 3 wks
- Release cadence
- Quarterly → continuous
- Production incidents per quarter
- 9 → 3
- Test coverage
- Manual scripts → automated eval + adversarial suite
Business impact
What it changed
71% faster releases
Average time from a model update being ready to reaching production dropped from 10.5 weeks to 3 weeks, once release authority moved from a quarterly board to an automated gate.
Incident rate fell alongside the faster cadence
Production incidents dropped from 9 to 3 per quarter — faster releases came with fewer failures, not more, because every update now clears the same eval bar.
Issues isolated to a single update
Because updates no longer ship in a quarterly batch, a regression can be traced to the specific change that caused it instead of a quarter's worth of candidates.
Technology stack
“Shipping less often was never the safety mechanism — the eval was. Once the gate did the checking, the calendar stopped being the constraint.”
Keep reading