Investigating Agent Quality

When an AI workload completes successfully, you know that the runtime configuration was correct, but not whether the output is at a level of quality required. This page discusses how we can investigate agent quality using evaluation tools. Look at agent evaluations to learn more about which tools are available. One thing you will notice is that evaluation is not a one-tool-fits-all sort of thing. We discuss it how you need pass/fail and directional tests for AI workloads to really succeed.

Investigating agent quality starts with looking at how individual runs are performing. Since Jetty captures all the inputs and outputs, it's easy to use Jetty to investigate how a single run went and then how to improve it. The same strategy can be applied to some common use cases:

• Surfacing AI costs
• Identifying LLM errors
• Picking the best model for the job

Once you are getting results, you can share them with your team or schedule the workload to run regularly.

Start with the run

Every execution is captured as a run: a complete, replayable trace of each step, the inputs it received, the outputs it produced, the files it wrote, and how long it took. When a result looks wrong, you don't guess at what the agent did; you open the run and read it. The log shows which tools and skills the agent reached for, and the step boundaries show where the work went sideways.

Investigating the results

Identify LLM errors

Most agent harnesses are resilient to failure; if something doesn't work the first time, they will try and find an alternate route. For instance, if accessing the web through a terminal command fails, they may spin up a web browser and navigate to the page instead. Jetty logs every step and those misteps can be used to improve the agent runbook.

Surface costs

The bulk of AI spend often comes from the cost for tokens. AI gateways and model provider costs vary wildly and they are often time a proxy for the reasoning performance of a model. Jetty makes it easy to take an AI workload and measure it across multiple models. The evaluations (code checks, checklists and judges) provide the quality dimension. Cost, time to completion and token consumption provide the other half of the equation.

Picking the best model for the AI workload

With investigations suited to your task, you can now make an informed decision and share the results with your team.

Sorting the cases by judge-scored complexity shows where each model's success rate falls off. The cheapest model whose reach covers your cases is usually the right one.

Add a check at the failure moment

Most evals get written after a run disappoints, not before, and that's fine. When a run fails in a way nothing caught, use the agent skill to modify the runbook with an evaluation: a code check if you can assert it, a checklist item if the agent stopped early, a judge criterion if it takes a reader.

Before you rely on a new code check, run it against the outputs of a few recent passing runs. A failed code check fails the run, so a buggy check fails every run and takes your trust in the whole eval with it. If the passing runs fail your new check, the check is wrong, not the runs.

Budget the instruments by cost. Code checks and the checklist are free to run every time, in flight. Judges cost tokens: score every run with a cheap LLM judge if you like, and send a sample to an agent judge for the verdict you trust.

Investigate and improve

Investigation only matters if it changes the next run. Because every run is recorded in full and every runbook carries its evals, the loop is short:

1. Run the runbook over a few cases. Each run records a verdict.
2. Read the runs that failed. The trace shows which step fell short and which check caught it.
3. Edit the runbook: add the missing check, tighten a step, fix a threshold.
4. Re-run and confirm the verdicts improved. Repeat.

optimize-runbook is a skill in the Jetty plugin that runs this loop for you. Point it at a runbook and it reads the prior runs, finds the failure patterns, proposes targeted edits (surgical changes, not a rewrite), and re-runs the evals to confirm the change actually helped before you keep it. No plugin? Any MCP-connected agent can do the same by hand with the MCP server: list-trajectories and get-trajectory to read the runs, update-task to apply the edit, and run-workflow to re-check.

Keep scores honest

A rising average is not proof the workload got better; it might mean the judge got easier to please. Read the trend across iterations rather than a single number. And keep checking the judge against a person: when you disagree with a verdict, label the run with your own, and treat agreement between your labels and the judge's as the health metric for the eval itself. If that agreement drifts, fix the judge before you trust another score from it.