Get Started
Documentation menu

Improving agent runbooks

The first version of a runbook is a hypothesis. It says what you think the agent needs to be told, and what you think “done” looks like. The runs are the evidence. This guide is about closing that gap on purpose: reading the last few runs, tightening the evaluations they expose, and editing the runbook so the next run is measurably better. It is the hands-on companion to Investigating Agent Quality, and it assumes you already have a runbook that runs (see Writing runbooks if not).

There are two places to do this work. The Jetty agent skill in your own coding agent is the fully supported path and the one this guide follows. Jetty Web also has an Optimize view that runs the same analysis against a single run and shows the proposed edits as a diff; it is currently in beta, so expect rough edges, and use the skill whenever you want to look across several runs or keep the file under your own editor's control.

The loop

  1. Pull the runs. Ask your coding agent to fetch the most recent runs for the runbook: their verdicts, their output files, and the agent's own log.
  2. Improve the evals first. For every miss the runs show, ask which check would have caught it, and add that check to the evaluations the runbook already defines.
  3. Then improve the instructions. Fix the step that produced the miss: a missing guardrail, an ambiguous instruction, a Common Fix the agent needed and didn't have.
  4. Re-run and compare. Two or three runs, side by side with the ones you started from. Keep the change if the verdicts moved; revert it if they didn't.

The fastest loop: the optimize-runbook skill

With the Jetty plugin installed, the optimize-runbook skill runs the loop above for you. Point it at a runbook and a task and it fetches the last N runs, reads each one's status, duration, step outputs, errors, and validation_report.json, then looks for six patterns:

  • Consistent failures: a criterion or stage below threshold in more than half of the runs.
  • Iteration waste: steps that need two or three retries every time.
  • Timeouts and bottlenecks: runs that ran long, and which step ate the time.
  • Divergent behaviour: structurally different outputs from the same instructions.
  • Missing guardrails: errors the evaluation section never caught.
  • Score plateaus: rubric criteria that iterate without improving.

For each pattern it proposes a specific edit, quoting the exact current text and the exact replacement, and cites the runs that motivated it. You choose to apply all, pick individually, or save the analysis as a report. Applied edits bump the runbook's version so the deployed copy and the file on disk stay distinguishable.

/optimize-runbook acme/triage --last 5
/optimize-runbook acme/triage --trajectories 2ce3ec8e,d02e276b

Three or more runs give it real patterns; one run gives it one data point. If you have only one, read it yourself first (the next section), then run the skill once more runs exist.

Best practice: start from the runs, and improve the evals first

The single most useful habit is to ask your coding agent to pull down the most recent runs from a runbook and use them as the basis for updating the evaluations that are already defined. Not to write new instructions from memory, and not to guess at what went wrong: to read what actually happened and make the checks catch it next time.

Pull the last 5 runs of acme/triage. For each one, read validation_report.json,
summary.md, and the primary output. List every place a run fell short that no
evaluation stage caught. For each, propose the check that would have caught it,
added to the evaluation section that already exists in RUNBOOK.md. Don't rewrite
the instructions yet.

Why evals first: an instruction change without a check is a hope. A check without an instruction change still fails loudly, which is exactly what you want while you work out the fix. And a check written against a real miss is specific in a way a check written on day one never is. The evaluation styles all take this well: a new code check, a new checklist line, or a new judge criterion.

Before you keep a new code check, run it against the outputs of a few recent passing runs. If they fail it, the check is wrong, not the runs.

What to read in a run

Your agent can fetch all of this with the jetty skill or the MCP tools (list-trajectories, get-trajectory, and the file download endpoint). Ask for, in this order:

  • The verdict. validation_report.json: which stages passed, which failed, and whether overall_passed is honest about it.
  • The deliverables. The files named in primary_outputs. Open them. A stage can pass while the output is still wrong in a way you would spot in ten seconds.
  • The summary. summary.md: what the agent thought it did, and any limitations it admitted to.
  • The agent's own log. The tool-by-tool timeline shows where the minutes went and where the agent improvised. A run that took 25 minutes because 19 of them were spent retrying one tool call looks identical to a slow run until you read this.
  • Duration and cost. Compare against the step's timeout_sec. A run that finishes a minute under its timeout will not finish next week.

Lessons from the field

These are the changes that came out of real optimize passes. Each one started as a run that completed, looked fine at a glance, and wasn't.

“Completed” is not “passed”

A run can complete with a green status and zero output files, because the agent hit an error and stopped politely. Treat completed with no results files as a failure, and make overall_passed in validation_report.json the signal you trust. Give it a PARTIAL category for degraded-but-usable runs (a fallback data source, a skipped optional input) so it never has to lie in either direction.

Put the deterministic work in a script, and keep the agent's judgement small

If a runbook queries an API, computes numbers, and renders a report, the queries, the arithmetic, and the rendering should be a script the agent runs, not steps the agent improvises. Runs become fast, cheap, and comparable, and the agent's effort goes where it is worth something: reading the numbers and writing the two paragraphs that need a person. A runbook that produces a report every day should produce the same report every day except for the sentences that need thought.

Never make the agent retype a big file

An early version of one report runbook inlined a 73 KB builder script in a code block and told the agent to write it to disk. The agent spent 19 of the run's 25 minutes failing to do that, then pasted it in four chunks. Moving the script out of the runbook and mounting it as a file (upload it once, reference it in the task's file_paths, copy it from /app/assets at run time) cut the same run to two minutes. The runbook shrank from 91 KB to 20 KB, which also matters because the instruction has a hard size ceiling after parameter substitution, and past it the agent fails without a useful error. Keep runbooks small; keep code in files.

Make the checks structural and cheap

The checks that catch the most are the boring ones: two counts that must add up to a third, no unfilled template slot left in the output, no None in prose, no secret-looking string in a report, a note that cites at least one number per bullet, a file over a minimum size. They cost nothing to run every time, they never disagree with themselves, and each one encodes something a person once had to notice by eye.

Turn every real failure into a Common Fix

When a run fails on something specific (a query dialect quirk, a reserved word, a certificate bundle missing off-sandbox, a rate limit), the fix goes into the runbook's Common Fixes table as a row: the exact error text, the exact fix. The next run applies it in one retry instead of three. The optimize-runbook skill prefers these additive edits for a reason: they cannot make anything else worse.

Watch the clock

Read the agent duration on every run, not just the failed ones. When a run brushes its timeout, fix the cause first (usually one slow tool call or a retry loop) and only then raise timeout_sec. Raising the timeout on its own turns a fast failure into a slow one.

Keep the verdict honest

Bound every retry loop (three rounds is the convention) and make the runbook report what it could not fix rather than ship it. Write rubric criteria to describe the target, not the current output, because a rubric that describes what the runbook already produces will rubber-stamp it. When you disagree with a judge, label the run with your own verdict and watch the agreement rate; when it drifts, fix the judge before trusting another score.

Version, redeploy, and check that you did

A deployed task carries its own copy of the runbook. It is easy to improve the file on disk and leave version 1.0.0 running on the schedule. Bump the version on every structural change, redeploy, and read the deployed instruction back before you trust the next run.

Then let it run

Once two or three consecutive runs pass with the tightened evals, put the runbook on a cadence with Scheduling AI Workloads and keep reading the verdicts. Models and providers drift; a check that passed for a month and starts failing is the runbook telling you something changed underneath it. That is the moment to pull the last few runs and start this loop again.