Runs

Every time Jetty runs something, such as a runbook, it records a run: a complete, replayable trace of what happened. This is the observability layer, and it is also the raw material that evaluations read when they evaluate an AI workload.

Starting a run

A deployed runbook can be launched from the web app, from your own agent through the Jetty plugin, from code with the SDK or the API, or on a cadence with a schedule. Every entry point fires the same workload, so runs from all of them land in one history and can be compared like for like. The Run includes the following components.

The parts of a run

• Inputs. The parameters the run was launched with and any uploaded files, frozen at launch. Uploads are mounted at /app/assets/.
• Runtime configuration. The agent, model and provider, the sandbox snapshot and its resources, and the MCP servers, snapshotted into the record. Secrets are never stored.
• The runbook, as merged. The exact instruction the agent read, with variables substituted and uploaded files listed.
• Outputs and files. Everything the agent wrote to /app/results/, with the frontmatter's primary outputs surfaced first. A run that ends with an empty results directory fails.
• Logs. The step-by-step workflow log, streamed live while the run is going and kept afterwards, plus the agent's own transcript.
• Telemetry. Duration, tokens in and out, API calls, and an estimated cost, recorded per step.
• Evaluations. The validation report the agent writes as it checks its own work: which stages passed, the rubric scores, the threshold, and how many fix-and-recheck iterations it took. See evaluations.
• Labels. Key/value tags added after the fact, by you, by Pelly, or through the API.

For each step, the run also records the activity that ran and how long it took, the inputs it received (resolved from path expressions), the outputs it produced, and its status (running, completed, failed, cancelled).

Runs are immutable historical records. You can replay one in the app's run viewer or using the API.