Runs

Every time Jetty runs something, such as a runbook, it records a run: a complete, replayable trace of what happened. This is the observability layer, and it is also the raw material that evaluations read when they evaluate an AI workload.

Starting a run

A deployed runbook can be launched from the web app, from your own agent through the Jetty plugin, from code with the SDK or the API, or on a cadence with a schedule. Below we detail the lifecycle of each runbook run. Every entry point fires the same workload, so runs from all of them land in one history and can be compared like for like.

The run lifecycle

A run is a closed loop that the agent performs with a predefined set out output criteria. At launch, the inputs and the runtime configuration are handed to the agent with the runbook inside a sandbox. The agent works in /app and produces a draft. The checks compare that draft against the output criteria. When the agent passes the evaluation gate, it ships the results, which are persisted and surfaced on the run. A fail sends the agent back to fix and recheck, and that loop is bounded. Either way the run is recorded, with its logs, telemetry, validation report, and any labels added afterwards. The Run includes the following components.

The parts of a run

• Inputs. The parameters the run was launched with and any uploaded files, frozen at launch. Uploads are mounted at /app/assets/.
• Runtime configuration. The agent, model and provider, the sandbox snapshot and its resources, and the MCP servers, snapshotted into the record. Secrets are never stored.
• The runbook, as merged. The exact instruction the agent read, with variables substituted and uploaded files listed.
• Outputs and files. Everything the agent wrote to /app/results/, with the frontmatter's primary outputs surfaced first. A run that ends with an empty results directory fails.
• Logs. The step-by-step workflow log, streamed live while the run is going and kept afterwards, plus the agent's own transcript.
• Telemetry. Duration, tokens in and out, API calls, and an estimated cost, recorded per step.
• Evaluations. The validation report the agent writes as it checks its own work: which stages passed, the rubric scores, the threshold, and how many fix-and-recheck iterations it took. See evaluations.
• Labels. Key/value tags added after the fact, by you, by Pelly, or through the API.

For each step, the run also records the activity that ran and how long it took, the inputs it received (resolved from path expressions), the outputs it produced, and its status (running, completed, failed, cancelled).

You can view a run in the app's run viewer or use Jetty's agent tools to access it for further investigation.