Getting ahead of new model updates: automated testing across agents

Taylor HodgeLead Engineer, Primitive

Primitive is building the decision layer for teams shipping code with agents. Primitive’s prim client plugs into the coding agents its customers already use: Claude Code, Cursor, Codex and others. But each agent behaves slightly differently, and each one can be updated on a different schedule. Since engineering teams are using a diversity of agents and models at the same time, Primitive needs to reliably deploy updates to prim that will support each member of their team and meet them where they work.

“When I use Jetty, I have a new level of trust in the behavior of the non-deterministic systems outside of my control.”

01 · Situation

Taylor Hodge is leading engineering, and with every customer bringing their own coding agent, he needed to know whether each new version of Primitive’s plugin would work as expected.

For example, a recent Codex update changed how life-cycle hooks worked, but nothing surfaced to their users. This caused the overall user experience to suffer.

Call, return, improve: the parent launches the same workload under each runtime configuration, the verdicts and their evidence come back to one report, and each improvement goes into the next sweep.

02 · Solution

Primitive’s continuous integration tools already ran synthetic prompts against live providers and used Jetty as a judge on the outputs. Taylor wanted more than that: every agent, and every model, taken through Primitive’s whole workflow end to end, with a clear account of where things stop matching expectations.

Primitive deployed a grader runbook that scores single eval cases, and its source lives in Primitive’s repository. The sweep is a runbook that runs runbooks. One parent run launches the cell runbook once per agent and model, then gathers every verdict into a single investigation report.

Jetty provided a delivery workflow that was easy to scale to new runtime configurations:

Same runbook, different inputs: only the agent, the model and the prim version change between runs, so a difference points at the agent, not the test.
Evals at two levels: each child run scores the agent on Primitive’s journey checks; the parent’s own checks fail the investigation if any run goes unreported or any failure goes unexplained.
Evidence, not opinions: a run that ends without a report is relaunched once, and every diagnosis in the report has to quote the runs’ own output.

A step that runs runbooks: the parent’s second step launches the same child runbook for every agent, and the verdicts come back to step 3.

03 · Outcome

The runbooks deployed on Jetty were able to catch silent failures before they degraded customer performance. Better still, the team was able to proactively report and review complex workloads across a battery of model and agent configurations, and where necessary, deep dive into individual runs to improve the plugin integration. The investigation runbook opens with a single grid of every check against every agent, then gives each failure a diagnosis quoted from the runs.

Because the sweep is itself a runbook, it reruns in parallel and on demand: a dozen runs across several agents with results in minutes.

How a sweep reads: one parent launches the same child runbook for each agent, three runs each. Every square is one check in one run; outlined clusters are diagnosed once. Illustrative agents and results, not Primitive’s data.