Investigating Agent Quality

Running a workload tells you that the agent finished. It doesn't tell you whether the output is any good. Jetty treats that second question as a first-class part of the product: every run leaves behind the evidence you need to answer it, without instrumenting anything yourself.

Start with the run

Every execution is captured as a run — a complete, replayable trace of each step: the inputs it received, the outputs it produced, the files it wrote, and how long it took. When a result looks wrong, you don't guess at what the agent did; you open the run and read it.

Read the workload report

When a run completes, Jetty grades it against the standard the runbook itself declares — the output manifest and the evals. The result is a workload report: which checks passed, which failed, and the scores behind them. Two kinds of checks feed it:

• Programmatic checks — deterministic assertions: the required files exist, the fields are present, the numbers are in range.
• Rubric scoring — a second model judges the output against named dimensions, passing only above the thresholds you set.

Compare runs, not impressions

One good run proves little; agents are noisy. Labels let you tag runs by configuration, cohort, or verdict and compare across them — did the new prompt actually raise the pass rate, or did you get a lucky sample? Because every run is scored the same way, the comparison is apples to apples.

Close the loop

Investigation only matters if it changes the next run. The evaluating & optimizing guide covers the full loop: read the failed checks, trace them to the step that went wrong, amend the runbook, and re-run — with /optimize-runbook automating the read-the-runs-and-propose-fixes step. Over time the runbook hill-climbs: the evals stay fixed while the pass rate rises.