Code Checks

A code check tests a run's outputs and answers pass or fail. It is written in the runbook, under ## Code Checks. There are two kinds, told apart by who runs them, and a run takes them in this order:

• Agent checks are run and reported by the agent while it works, for what only the agent can reach.
• Jetty checks are run by Jetty after the agent has finished, so the agent cannot skip them, change them or misreport them.

Both count the same in the verdict. Prefer a Jetty check wherever a command can see what has to be checked: its result does not depend on the agent under test.

Writing a check

Each check is a level-three heading followed by one fenced block. The heading carries an id and a name separated by a dash. The fence's language says what the check is, and an executor= word after it says who runs it. One check of each kind, in the order a run takes them:

## Code Checks

### issue-filed — The Linear issue exists and is Done
```agent executor=agent
Use the linear MCP server to fetch the issue whose key is in
{{results_dir}}/issue.txt. Pass if it exists and its state is Done;
fail otherwise, and say which state it is in.
```

### api-up — The API the agent started answers
```bash executor=agent
curl -fsS http://localhost:8000/health
```

### outputs-exist — Every required output is present
```bash executor=jetty
test -s {{results_dir}}/summary.md && test -s {{results_dir}}/output.json
```

### output-is-json — output.json parses
```yaml executor=jetty
use: json_valid
path: output.json
```

executor=agent is the only choice for an agent fence and executor=jetty the default for a command, so both can be left off. The fence lines Jetty accepts:

• The id is letters, digits, ., _ and -. It names the check in the report and the eval view, so keep it stable. The name is free text: say what passing means.
• Only the first fence under a heading counts. executor= is case-insensitive, and any other word on the fence line is ignored.
• A check that cannot run is recorded as error with the reason, never silently dropped: a heading with no usable fence, an unknown language or built-in, an empty command, a {{checks_dir}} path for an undeclared source, a template blank such as {TODO: …}, ```agent executor=jetty, a misspelt executor (executor=jety, executor:agent), or ``` executor=agent with no language.

Agent checks

Instruction checks

Some facts cannot be read from a run's files: the issue exists in Linear, the page renders in a browser, the row landed in the database. Checking them takes a tool, and an MCP server in particular needs an MCP client, which only the agent has. Write the check as an instruction in an agent fence: say what to do, with which tools, and what passing means.

The MCP servers a check needs are declared in the frontmatter, under mcp_servers or code_checks.mcp_servers. A definition never carries a token: it names a secrets: entry with secret:, and Jetty puts the value into the server's configuration when the run starts (see Secrets). A credential-looking key written into a definition is dropped with a warning. Authenticated MCP servers reach the agent on the claude-code, gemini-cli and antigravity runtimes.

Commands the agent runs

A command that needs the agent's live session, such as a server the agent started, is marked executor=agent and run by the agent before it reports. It names paths with the same {{results_dir}}, {{checks_dir}} and {{assets_dir}} placeholders as any other check; Jetty fills them in before the agent reads the runbook. It runs with the agent's environment, which does not have the variables Jetty gives its own checks.

### summary-written — summary.md is not empty
```bash executor=agent
test -s {{results_dir}}/summary.md
```

• A command that reads a secret the agent does not receive is refused before the run, as an error.
• When one uses {{checks_dir}}, Jetty clones the git sources before the agent starts.

How agent checks are reported

Jetty tells the agent the id of every check it must run, and the agent records one code_check entry per id in its report, with pass or fail and a one-line message. Jetty does not re-run it. An entry for an id the runbook never declared is dropped, and a declared check the report leaves out is recorded as skipped. See how the report comes together.

Jetty checks

How Jetty checks run

Jetty checks are not part of the agent's fix-and-recheck loop. The agent is told that Jetty runs them after it finishes and not to run them itself, so it never sees their result. A failed Jetty check shows in the run's verdict, not in another round of work.

Once the agent has finished, Jetty stops every process it left behind, so nothing can change the outputs, and removes its credential files. It restores the uploaded assets from its own copy, clones the git sources fresh, and runs each check in turn in the sandbox the agent worked in, against the outputs exactly as the agent left them: the same files Jetty saves. Each check gets a profile-less shell with the PATH the sandbox had before the agent ran. It sees the outputs, the assets, the sources and the secrets given to the checks, and not the agent's model credentials, MCP servers or session.

Inline commands

The fence is the command. It runs from /app in bash with -e and pipefail set, so every line of a multi-line fence must succeed. Exit 0 is pass; any other exit is fail, except the ones that mean the check could not be carried out (see Results). The last non-empty line of output becomes the check's message.

Paths are written with the {{results_dir}}, {{checks_dir}} and {{assets_dir}} placeholders, and the run's own {{name}} inputs are filled in as they are in the rest of the runbook. A run cannot change these paths. For a script that reads them from its environment, a Jetty check also has them as variables:

Built-in checks

The common cases need no code. A yaml fence names a built-in with use: and its parameters, all required. Paths and globs are relative to the results directory; a pattern is a Python regular expression, matched line by line. Missing or unusable parameters are an error, not a failure.

How Jetty checks are reported

Jetty adds one code_check entry per check after the agent's entries, marked as run by Jetty, with the command, the exit code, the last 2 KB of output and how long it took. code_checks.json keeps up to 16 KB of output for each. An entry the agent wrote for a Jetty check is dropped, and if the sandbox could not be prepared, every Jetty check is an error naming why. See how the report comes together.

Bringing code with the check

When a check outgrows one line, keep it as a script and point the check at it:

• Upload it with the task. Files attached to the task are at /app/assets: python3 {{assets_dir}}/validate.py {{results_dir}}/output.json.
• Declare a git source in the frontmatter. Jetty clones it to /app/checks/<name> at the named branch, tag or commit (the default branch otherwise). The URL is a plain https:// address. A private repository names the secret to clone with; the token goes to git alone and never appears in the clone, the agent's environment or any log.

---
secrets:
  GITHUB_TOKEN: {}                 # clones the source below
  LINEAR_TOKEN: {}                 # authenticates the linear server
mcp_servers:
  linear:
    url: https://mcp.linear.app/sse
    secret: LINEAR_TOKEN           # names the secret, never the token
code_checks:
  timeout_sec: 120                 # per check; default 120, max 900
  sources:
    - name: checks                 # → /app/checks/checks
      type: git
      url: https://github.com/acme/output-checks
      ref: v1.4.0                  # a branch, a tag or a commit
      secret: GITHUB_TOKEN         # only for a private repository
---

These keys are defaults. When the runbook is saved as a task, Jetty copies mcp_servers (with code_checks.mcp_servers folded in), code_checks and strict_evaluation into the task's init_params. A run may override any of them, one top-level key at a time: a run that passes code_checks replaces the whole block. secrets is never copied: it declares which values the run must bring.

Jetty confirms every source is reachable with its credential, at its ref, before the sandbox starts, so a wrong URL, a missing ref or an expired token fails the run in seconds. For Jetty checks the clone happens after the agent has exited, so the agent cannot have changed the check code.

Secrets

A runbook names the secrets a run needs under secrets:; it never holds their values. Each value comes from the collection's environment variables, or from the run's secret_params, and is never stored with the task. An entry can say which environment variable the value arrives in, and who receives it:

secrets:
  OPENAI_KEY: {}                   # env var OPENAI_KEY; the agent receives it
  PUBLISH_TOKEN:
    env: GH_PUBLISH_TOKEN          # the env var it arrives in (default: its name)
    expose_to_agent: false         # keep it out of the agent's environment
    expose_to_checks: true         # but give it to the checks Jetty runs

• A declared secret the run does not supply is named in the run log, and the run goes on without it. An MCP server whose secret is missing is configured without a token; a source whose secret is missing usually fails the reachability check, with a hint naming it.
• A run may pass a server's token directly, under mcp_auth_tokens ({"linear": "…"}); the secret is then not used for that server.
• The value of every declared secret the run set is scrubbed from each check's output before it is kept.
• Keeping a secret out of the agent's environment is hygiene, not a security boundary: Jetty checks run in the sandbox the agent had, so a secret given to them is trusted to that sandbox.

Results

Each code check ends with one of four statuses:

fail and error fail the run's verdict, and so does skipped under strict evaluation. See the verdict.

Tips

• Test a new check against a run that passed. A wrong check fails the verdict of every run from then on, and you stop trusting the verdict.
• Print the reason. test -s file || echo "summary.md is empty" reads better in the eval view than a bare exit code.
• Keep checks fast. They run one after another, each under its own timeout. A check that needs the network or a slow tool is usually better as a script in a source repository, where it can be tested on its own.
• Do not depend on what the agent installed. The sandbox image and the runbook's pip_packages / apt_packages are what a check can rely on; python3 and git are on the default image.