Get Started
Documentation menu

Agent Evaluations

A finished run tells you the agent stopped. An evaluation tells you whether the work is any good. Jetty treats evaluation as a step in the workload rather than something bolted on afterwards: the checks live in the runbook, they run inside the sandbox while the agent is still working, and every run comes back with a verdict.

Why evals run in flight

There is no offline benchmark for “the receipts our company will receive next quarter.” Real workloads have no golden dataset, and you can't anticipate every new kind of input, so the evaluation has to travel with the job and run every time the job runs. The evaluations you add to a runbook enable improving AI workloads through experience. Jetty supports the following evaluations:

  • Code checks: code that runs against the outputs and returns pass or fail.
  • Checklists: a plain list of conditions the agent walks through before it reports done.
  • Judges: a model or an agent handed the output and a criterion, returning a verdict with evidence.

Code checks

A code check is code executed against the run's outputs. It returns pass or fail and nothing in between: the required files exist, the spreadsheet has the expected columns, every date parses and falls inside the window, totals reconcile, no receipt id appears twice. The agent runs them inside the sandbox as one of the runbook's steps, so a failure is caught while the agent can still fix it.

Code checks are deterministic, cheap, and need no model, so they never flake and can run on every run. They also sit at the top of the hierarchy: a failed code check fails the run, whatever a judge says about the same output. That authority cuts both ways: a wrong check fails every run, which is why a new check gets tested against passing runs before anyone trusts it.

Checklists

A checklist is a plain list of conditions the agent walks through before it reports done, written in the runbook next to the steps. Each item is small and observable: the summary names every flagged receipt, the CSV has one row per input image, the report links to its sources. The agent ticks through the list, fixes what fails, rechecks, and only then writes its report.

The checklist and the code checks run together as the runbook's evaluate step, followed by a bounded fix-and-recheck step (the templates allow three rounds), after which the agent writes a validation report with an overall pass or fail.

LLM and agent judges

A judge is handed three things: an output to assess, an instrument to apply, and optionally the source context. The instrument can be a yes/no question, a bare criterion, or a 1–5 scale, and a rubric is a set of these with thresholds. The judge returns a structured verdict with cited evidence, so the answer is both a value and an explanation: “out of range: a 40% tip on a $12 bill.” A judge written in English is executable, because the model can read. Judges are for the quality you can describe but can't assert in code: tone, completeness, whether an answer addresses the question that was asked.

Judges come in two shapes, and the difference is cost and depth.

  • LLM judges are a single model call. They are fast and cheap enough to run in flight: the runbook calls a judge model through the credentials minted for the run, hands it only the output and the criterion, and gets a verdict back. Because the context is deliberately narrow, a capable agent can be paired with a much smaller judge model. The simple_judge action is this shape as a workflow step, scoring 1–5 or against a categorical rubric.
  • Agent judges are a separate agent run, usually a grader runbook. The grader has tools: it opens the output files, checks them against sources, and works through the rubric the way a careful reviewer would. It is slower and costs more, and it is the more trustworthy verdict. The agent under test never sees it. This is the shape the Jetty and Friends examples use to grade agents running outside Jetty.

A runbook can also carry its own rubric and have the agent score itself against it, which is what the rubric template does: criteria on a 1–5 scale, an overall threshold, no criterion below 3, then up to three rounds of improving the weakest score. The research literature calls self-grading unprincipled, and it works surprisingly well as an in-flight discipline that keeps the agent honest about its own output. It is not the score you trust.

Keep the judge independent of the agent under test. When a weak model grades its own output, it scores its mistakes as successes, so every improvement you make is measured against a lie. The verdict that counts comes from a separate, capable model or from a code check. Judge ≠ subject.

Every run gets a verdict

Run the runbook and the result, a pass or fail from the checks and a score from any judge, is recorded on the run that produced it. Grading steps write the verdict as a label, so runs can be queried and sliced by verdict, configuration, model, or input cohort later. That recorded verdict is the raw material for investigation, which only works if the evals are in the runbook.