Judges

A judge is handed an output to assess, an instrument to apply (a yes/no question, a criterion, or a 1–5 scale) and optionally the source context. It returns a structured verdict with cited evidence, so the answer is both a value and an explanation: “out of range: a 40% tip on a $12 bill.” A rubric is a set of these instruments with thresholds.

At a glance

Judge

Quality that can be described but not asserted in code: tone, completeness, whether an answer addresses the question that was asked.

Run by

Agent, model or grader

Self-scored, a judge model, or a separate run

When

In the run or after

Rubric in the evaluate step; a grader as its own run

Fix-and-recheck

Rubric only

A separate grader is outside it

If not reported

No entry

Rubric criteria are not declared checks

Written as: A rubric table in a runbook written with evaluation: rubric, the simple_judge step, or a grader runbook.
In the report: A judge entry per rubric criterion, with its score, a max_score of 5 and a threshold of 3.

Example

### Rubric

| # | Criterion    | 5 (Excellent)             | 3 (Acceptable)  | 1 (Poor)       |
|---|--------------|---------------------------|-----------------|----------------|
| 1 | Accuracy     | Every figure has a source | Minor slips     | Wrong totals   |
| 2 | Completeness | Every receipt covered     | One or two gaps | Parts missing  |
| 3 | Clarity      | Reads cleanly             | Understandable  | Hard to follow |

**Pass threshold: overall average >= 4.0, no individual criterion below 3.**

Writing a rubric

A runbook written with evaluation: rubric carries its rubric as a table of criteria, each scored 1–5, with what 5, 3 and 1 look like, and a pass threshold (see rubric and programmatic evaluation). Write each column so two readers would give the same score: name what is present or missing, not how good it feels.

How judges run

Self-scored rubric. The agent scores its own output against the rubric in its evaluate step. Below the bar, it reworks the lowest-scoring criteria and scores again, up to three rounds, then keeps its best version and notes what is still weak.
LLM judge. A single model call, fast and cheap enough to run in flight: the runbook calls a judge model through the credentials minted for the run, hands it only the output and the criterion, and gets a verdict back. Because the context is narrow, a capable agent can be paired with a much smaller judge model. The simple_judge action is this shape as a workflow step, scoring 1–5 or against a categorical rubric.
Agent judge. A separate agent run, usually a grader runbook, with tools: it opens the output files, checks them against sources, and works through the rubric the way a careful reviewer would. It is slower and costs more, and the agent under test never sees it. The Jetty and Friends examples use this shape to grade agents running outside Jetty.

How scores are reported

Each rubric criterion becomes a judge entry in the agent's report. An entry whose score is below its threshold fails the run's verdict. The overall average is checked by a checklist item (“Overall rubric average is at least 4.0”), and the run's evaluation records it as overall_score against a pass_threshold of 4.0. See how the report comes together.

Tips

Treat a self-scored rubric as the agent's opinion. Agents love giving themselves a pat on the back. Back the criteria that matter with a Jetty check where a command can see them, and with a separate agent judge where it takes a reader.
Give the judge only what it needs. The output and the criterion, not the agent's whole transcript: a narrow context keeps the verdict about the output.