# Jetty — full documentation > Every page of the Jetty developer docs as Markdown, in sidebar order. The index of these pages, with one-line summaries, is https://jetty.io/llms.txt; any single page is also available as Markdown by requesting it with `Accept: text/markdown`. --- Source: https://jetty.io/docs/get-started # How to start with Jetty: bring your first workload Jetty is a platform for running and evaluating AI workloads. You describe a job in a runbook, a configuration file that defines the task, and the agent and model will run the task, with jetty checking whether the result is good. Jetty runs that job in a managed sandbox, scores the output against your checks, and keeps a full record of every run so you can compare one attempt against the next. This walkthrough covers the two fastest ways to bring a first workload into Jetty: the web app's built-in authoring tool, and your terminal, through the coding agent you already use or a plain shell client. Both produce the same kind of tracked, evaluated run. [Get Started Create an account go from sign-up to your first evaluated run in minutes. Create Account](/sign-up) ## What you need before you start - A Jetty account. The [free Scoop plan](/pricing) covers free runs. - Optional: API keys for whichever AI Gateway you want runs to use. Your first runs can use Jetty's own gateway. - For the terminal path: a coding agent that can fetch a URL and run shell commands, or git, curl and jq for the shell client. ## Path 1: the web app's authoring agent When you sign up, Jetty walks new accounts through a short three-step tour of the dashboard. The tour itself is worth knowing about before you start, because it tells you both things this guide covers: your dashboard is where every run lives once it's underway. You can start your first runbook either from the Author Runbooks screen in the web app. ![The Author Runbooks screen in the Jetty web app: a New Runbook button and the list of authoring sessions.](/_next/image?url=%2Fdocs%2Fget-started%2Fauthor-runbooks.png&w=3840&q=75) The Author Runbooks screen, from the dashboard sidebar. Author Runbooks is a single prompt box. The heading "Build runbooks using the Jetty authoring agent," asks you to describe a task you need done reliably and repeatably, with an example placeholder along the lines of "Generate a competitor analysis report." The agent behind the box has a name: Pelly, after Jetty's pelican mascot. The onboarding tour explains the mechanism plainly: tell Pelly what you need done, and it turns that description into a repeatable runbook you can run anytime. ![The prompt box headed Build runbooks using the Jetty authoring agent, with the placeholder e.g. Generate a competitor analysis report and a submit arrow.](/_next/image?url=%2Fdocs%2Fget-started%2Fdescribe-task.png&w=3840&q=75) The prompt box: describe the task, then submit it. To try it: 1. From the dashboard sidebar, open **Author Runbooks**. 2. Describe the job the way you'd brief a person doing it for the first time rather than a prompt template. Write what the output should contain, not just the topic. Pelly has more to work with from "a one-page comparison of three competitors' pricing pages, as a markdown table" than from "competitor research." 3. Submit it. 4. Pelly opens a dedicated authoring screen built around four stages: **Define runbook → Run → Evaluate → Optimize**. The first stage is a back-and-forth, conversation: Pelly confirms the task back to you in a chat panel on one side, while a draft runbook fills in live on the right pane, a rendered preview of the runbook is filled in. Reply in the chat to correct or add detail, the draft updates as you go. 5. Once the instructions look right, move through Run and Evaluate, and from that point every run is tracked the same way regardless of how the runbook got written: inputs and configuration, the full agent transcript, and the evaluation results. ## Path 2: from your terminal If you'd rather work from your terminal, there are two ways in, and both end at the same tracked, evaluated run as Path 1. The quickest is to let the coding agent you already use install Jetty for itself. The other is a shell client you source into your own session, for scripting without an agent in the loop. **Let your agent install itself.** Jetty publishes its setup procedure as machine-readable instructions at [`https://jetty.io/agent-instructions.md`](/agent-instructions.md): numbered steps written for the agent rather than for you, with the points that need a human marked as such, and a report-back block at the end. Paste one line into Claude Code, Codex, Cursor, or any agent that can fetch a URL and run shell commands: Bringing your own agent? Guide me through installing https://www.jetty.io/agent-instructions.md and setting up my first runbook. [Full instructions](/docs/integrations/bring-your-own-agent#install-the-plugin) The agent reads the instructions, looks for an existing token at `~/.config/jetty/token`, stops only where it needs you (pasting an API key or signing up, and adding your own provider keys if you skip the free trial), stores the key, and hands off to the runbook wizard. The deeper reference it consults along the way is [`https://jetty.io/MACHINE_CONTEXT.md`](/MACHINE_CONTEXT.md). The same prompt sits on the jetty.io home page and on the [Bring Your Agent docs page](/docs/integrations/bring-your-own-agent), with a copy button. **Or install the plugin yourself.** In Claude Code, two commands add Jetty's plugin marketplace and install the plugin, which bundles the MCP server with four skills: jetty, jetty-setup, create-runbook and optimize-runbook. Then run the setup wizard once: Shell ``` claude plugin marketplace add jettyio/jettyio-skills claude plugin install jetty@jetty /jetty-setup ``` The wizard creates or links your account, writes your token to `~/.config/jetty/token`, and checks that your provider keys are in place. Codex and OpenCode have plugin manifests in the same repository. Editors that speak MCP but not skills, Cursor, VS Code, Windsurf, Zed, add the server on its own, `npx -y jetty-mcp-server` with `JETTY_API_TOKEN` in its environment, and point the agent at agent-instructions.md for onboarding. **The shell client, for scripting without an agent.** Jetty has no standalone binary yet. The shell client is a bash script in the same repository as the skills; it needs git, curl and jq: Shell ``` git clone https://github.com/jettyio/jettyio-skills.git source jettyio-skills/skills/jetty/jetty-cli.sh ``` Authenticate with an API key from Settings → API Keys in the web app (it starts with `mlc_`): Shell ``` export JETTY_API_TOKEN="mlc_your_token_here" ``` A handful of commands cover a first workload end to end. Your workspace holds your runbooks, and a task is a runbook deployed inside it. The CLI and API name a workspace `collection`: - `jetty_health`, confirms the client can reach your account - `jetty_collections`, lists your workspaces - `jetty_tasks my-workspace`, lists the runbooks deployed in a workspace - `jetty_run my-workspace my-task '{"vars": {"input_file": "sample.pdf"}}'`, starts a run and returns its workflow id; the runbook's parameters travel under `vars` - `jetty_trajectories my-workspace my-task`, lists recent runs with their status - `jetty_trajectory my-workspace my-task `, pulls one run's full record, steps and outputs included - `jetty_help`, the full command reference Start runs with `jetty_run` rather than `jetty_run_sync`: a synchronous call that outlasts about 100 seconds returns a gateway timeout while the run carries on in Jetty, so start the run, then poll it. Don't retry the start, a retry launches a second run. This path fits better once you're scripting a workload into something else, a CI pipeline, a scheduled job, your own internal tooling, rather than exploring interactively. ## Writing a runbook directly Both paths eventually hand you a runbook: one markdown file with frontmatter for the agent, model, and sandbox, an objective, steps, evaluations, and a declared list of required output files. The bare minimum, stripped down to just what's needed to run, looks like this: Markdown ``` --- agent: claude-code model: claude-sonnet-5 --- # Summarize a document ## Steps 1. Read {{input_file}} and produce a plain-language summary under 200 words. ## Checklist - Confirm the summary doesn't introduce any fact not in the source document. ## Required output files - results/summary.md ``` That covers the essentials, but it's deliberately pared down. What Pelly actually generates through Author Runbooks carries more frontmatter than this, a declared evaluation approach, an explicit sandbox image, a provider-routed model, and a secrets block for anything the job needs credentials for, plus a fuller Objective section up top and a table instead of a bullet list for required outputs. [Authoring your first runbook](/docs/guides/authoring-your-first-runbook), a companion guide, takes a draft like that to a passing run, with a full worked example, and the [Runbooks](/docs/ai-workloads/runbooks) page covers the parts of the file. ## Pick what counts as "done" Whichever path produced the runbook, the evaluation is what makes a run worth tracking rather than just producing output. Evaluations are written into the runbook next to the instructions, Jetty carries them out as part of every run, and their results add up to one verdict per run: pass, fail, or unknown when nothing was decided. The runbook's frontmatter names one of two patterns, `evaluation: rubric` for output whose quality has several dimensions (reports, summaries, images) and `evaluation: programmatic` for output with a right answer (structured data, extraction, code), and either pattern can carry three kinds of check: - **Code checks**, commands that give a deterministic pass or fail, in two kinds told apart by who runs them. An agent check is run by the agent itself, with its own tools, before it writes its report, so a failure gets fixed inside its fix-and-recheck loop; the result is the agent's word, and a check it leaves out is recorded as skipped. A Jetty check is a command or built-in check that Jetty runs in the sandbox after the agent has finished (the files exist, the JSON parses, the totals reconcile), out of the agent's reach, so it can neither skip it nor misreport it, and a failure goes straight to the verdict. - **Checklists**, plain-language conditions the agent confirms by inspecting its own output before reporting done, with a bounded number of fix-and-recheck rounds, three in the templates. Like agent checks, an item left out is recorded as skipped. - **Judges**, a model or agent handed the output and a criterion, returning a score with evidence: tone, completeness, whether an answer addresses the question. A rubric is the agent scoring itself, 1 to 5 per criterion, with a bar of no criterion below 3 and an average of at least 4.0; a separate grader run is the form the agent can't influence. Two facts to hold on to before the first run. One failed checklist item fails the verdict as surely as a failed Jetty check, there is no partial credit, and what differs between types is how far you can trust a pass. And a failed evaluation is not a failed run: the run completes and keeps its outputs while the verdict says the work fell short, and a run fails only when the job could not be carried out at all. Once a runbook's agent checks are settled, set `strict_evaluation: true` in the frontmatter so a skipped check counts as failed and an agent can't pass by leaving out the check it would fail. See the [evaluations](/docs/ai-workloads/agent-evaluations) reference for an overview: the two patterns, how the report comes together, the verdict, strict evaluation, and a failed run set against a failed evaluation. [Code Checks](/docs/ai-workloads/agent-evaluations/code-checks) covers agent checks and Jetty checks, writing a check, the code and secrets a check can use, and what it records. The [Checklists](/docs/ai-workloads/agent-evaluations/checklists) page covers writing one and how each item reaches the verdict. Finally, [Judges and Rubrics](/docs/ai-workloads/agent-evaluations/judges) covers LLM judges, agent judges and rubric scoring. [Defining Agent Checklists](/docs/guides/ai-agent-checklist) and [Building AI Evaluations](/docs/guides/building-ai-evaluations) are the guides for writing them. ## Read the result, then make incremental changes After a run finishes, you can review what happened. Every run keeps the exact runbook the agent read, a step-by-step log, the full agent transcript, and the validation report your checks produced. This report has details about which checks worked and how many fix-and-recheck rounds it took. From here, the loop is the same one Jetty is built around: author, run, investigate, improve. Change one variable, a model, a prompt line, a check, and compare the new run against the last one. [Improving Agent Runbooks](/docs/guides/improving-agent-runbooks) covers this once a first workload is running reliably. ## Frequently asked questions **What is Jetty used for?** Jetty runs and evaluates AI workloads, jobs handed to a coding agent that need checking for quality and tracking over time, rather than one-off chat requests. **Do I need to write code to try Jetty?** No. Through the web app, describing a task to Pelly in plain language is enough to get a runbook. The terminal path is still a one-line prompt to your agent or a few shell commands, no application code required. **What's the difference between the web app path and the terminal path?** The web app's Author Runbooks screen generates a runbook from a plain-language description through Pelly, Jetty's authoring agent, and is the fastest way to try something interactively. The terminal path brings Jetty to the coding agent you already use, either by letting the agent install itself from jetty.io/agent-instructions.md or by installing the plugin or MCP server, with a shell client for scripting a workload into a pipeline, a schedule, or other tooling you already run. **What's the difference between code checks, checklists, and judges?** Code checks are deterministic pass or fail commands, run by the agent inside its fix-and-recheck loop or by Jetty after the agent finishes, where the agent can neither skip nor misreport them. Checklists are observable conditions the agent confirms about its own work. Judges are model-based scoring for qualities that are hard to express in code. Most real runbooks use a mix of all three, and no type outranks another: any failed check fails the run's verdict, while the run itself completes and keeps its outputs. **How much does it cost to start?** The Scoop plan is free, no credit card required. Your first 12 runs through Author Runbooks are on us. See [full pricing](/pricing). **Where do I go after my first run?** Read the validation report, not just the pass or fail result, then change one thing and compare the next run against it. [Improving Agent Runbooks](/docs/guides/improving-agent-runbooks) and [Creating a Benchmark](/docs/guides/creating-a-benchmark) cover that loop once a first workload is past its first run. [See the full product overview](/docs/product-overview) or [start now](/sign-up). --- Source: https://jetty.io/runbooks # Runbook Directory — Ready-to-Run AI Agent Workflows · Jetty ## Featured runbooks28 Open one to read the full runbook, its example outputs, and how it's evaluated. [Slide Maker 8–15 mins Generate presentation decks grounded in real GitHub projects, or walk through a structured brief-to-slides process. ![](https://github.com/adewale.png)adewalegithubslides](/r/adewale/slide-maker) [PDF processing 1–2 mins Extract text, tables, and key fields from PDFs into clean, structured JSON. ![](https://github.com/anthropics.png)anthropicsfilespython](/r/anthropics/pdf) [Durable Objects 2–4 mins Turn a spec into a production-ready Cloudflare Durable Object in TypeScript, self-reviewed against the gotchas. ![](https://github.com/cloudflare.png)cloudflaretypescriptcloudflare](/r/cloudflare/durable-objects) [Cross-model benchmark 2–4 mins Run the same prompt through two or more models side by side and score the results. ![](https://github.com/garrytan.png)garrytanai-evalbenchmark](/r/garrytan/gstack-benchmark-models) [SEO audit 6–9 mins Crawl a site, score each page against on-page SEO checks, and rank the fixes by impact. ![](https://github.com/coreyhaines31.png)coreyhaines31webseo](/r/coreyhaines31/seo-audit) [Canary 3–5 mins QA a described user flow against a live web app — the agent drives a real browser and hands back a report plus a replay script. ![](https://github.com/wizenheimer.png)wizenheimerqabrowser](/r/wizenheimer/canary) [Conference abstract scraper 15-35 mins Scrape a conference's accepted abstracts into structured JSON, pairing CrossRef metadata with full text from the meeting program. smnthscrapingresearch](/r/smnth/conference-abstract-scraper) [Policy-based text redaction 15-20 mins Redact sensitive spans from each record in a dataset according to its own policy, then score the result against a five-point rubric. smnthprivacycompliance](/r/smnth/policy-based-text-redaction) [Complementary Vulnerabilities: Auditing and Governing Clinician-LLM Error Overlap 2-4 mins But a second reasoner reduces error only insofar as it fails differently from the first. We say that a clinician and a model have complementary vulnerabilities to the extent that they break down through different mechanisms and on different cases. We therefore propose a predeployment audit in which each site measures error overlap on its own cases. smash2026research papersllm-evaluation](/r/smash2026/complementary-vulnerabilities) [Gated Knowledge-Guided Prototype Transfer for Zero-Support Medical Vision Language Model 2-4 mins We introduce gated Knowledge-Guided Prototype Transfer (KGPT), a pre-repulsion module that transfers supported-class visual residuals to missing classes using VLM similarity and an ontology-based clinical distance. A validation-free evidence gate scales transfer based on semantic relevance and residual agreement, thereby limiting unsupported extrapolation. smash2026research papersclinical](/r/smash2026/gated-kgpt) [Query Logs Are Patient Records: An Unguarded Attack Surface in Clinical RAG, and a Privacy-Preserving Pipeline 3-5 mins In privacy-sensitive applications, the server observes the query and controls which evidence is returned, creating both confidentiality and integrity risks. Our protocol reduces private and verifiable retrieval to multiplication of a committed matrix by an encrypted vector and uses low-bit quantization to make this computation practical. smash2026research papersprivacy](/r/smash2026/query-logs-patient-records) [Visual Acuity Improvement Prediction in Neovascular Age-Related Macular Degeneration 2-4 mins We proposed a predictive pipeline that automates the extraction of clinical OCT biomarkers using fine-tuned DL models. The combination of these automated biomarkers with baseline best-corrected visual acuity serves as inputs for an interpretable classifier. smash2026research papersclinical](/r/smash2026/nAMD-visual-acuity) [From Canon to Nikon: Camera Domain Shift and Potential Equity Implications in Retinal AI 3-5 mins In a Brazilian cohort of 8,524 patients, retinal images acquired with the Nikon NF5050 were more than twice as likely to have inadequate image quality compared with those acquired with the Canon CR-2, despite refurbished Canon systems costing more than twice as much as Nikon systems. We find that camera-associated performance differences vary substantially across tasks, and that neither higher input resolution nor classifier choice consistently mitigates cross-camera variation. smash2026research papersclinical](/r/smash2026/canon-to-nikon) [Prevalence, Thresholds, and Subgroup Performance Gaps in Clinical Prediction Models Across Five Public Datasets 4-7 mins Clinical prediction models are commonly compared across demographic subgroups, but a raw performance gap is not a fixed property of a model. Observed gaps should prompt investigation of metric choice, prevalence, threshold, sample composition, measured case mix, and uncertainty rather than being treated alone as proof of discrimination, fairness, or equivalence. smash2026research papersclinical](/r/smash2026/prevalence-thresholds) [Implementing Ambient Artificial Intelligence (AI) Scribes in Hospital-based Specialty Care: A Mixed-Methods Evaluation on Administrative Burden and Clinical Workflows 2-4 mins Among 17 mid-pilot survey respondents, while 52% reported the AI scribe was easy to use, 88% reported that AI generated notes required substantial editing … These interim findings suggest that the implementation of an AI scribe is feasible in hospital-based ambulatory care but may not independently reduce documentation burden … smash2026research paperspeople](/r/smash2026/ambient-ai-scribes) [Diagnosing Low-Confidence Patient Profiles via Localized Harmful Covariate Shift Detection 2-5 mins MED3pa first discovers interpretable low-confidence patient profiles, and Detectron is then applied within each profile to test whether its degradation is associated with a detectable harmful covariate shift. An injected age shift, undetectable at the population level, was nonetheless localized to specific degraded profiles … smash2026research papersclinical](/r/smash2026/low-confidence-profiles) [TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation 2-5 mins TRACE learns a codebook of recurring activation patterns and assigns each token activation to its nearest codebook vector, defining a latent concept. During steering, activations assigned to harmful concepts are steered along directions defined by nearby benign targets. smash2026research papersllm-evaluation](/r/smash2026/trace-safety-steering) [MASQ: The Multi-Agent System Questionnaire 2-5 mins We introduce the multi-agent system questionnaire (MASQ), a three-stage screening framework that connects system design choices to plausible miscoordination risks and their observable manifestations. Stage 3 guides researchers in examining MAS execution traces for evidence of anticipated and unanticipated miscoordination. smash2026research papersllm-evaluation](/r/smash2026/masq-questionnaire) [Beyond Algorithms: Organizing Artificial Intelligence in Healthcare Through Expertise, Governance, and Knowledge Infrastructures 2-5 mins We develop a framework that conceptualizes sustainable deployment of AI as an emergent organizational property produced through expertise coordination, governance infrastructures, and knowledge infrastructures. Four propositions are advanced linking expertise coordination and organizational infrastructure to accountability and sustainability. smash2026research paperspeople](/r/smash2026/organizing-healthcare-ai) [Privacy-Preserving Text Inference for Health-Misinformation Classification and Semantic Retrieval 6-10 mins We design a privacy-preserving text inference framework based on CKKS homomorphic encryption for health-misinformation classification and semantic retrieval. The hybrid classifier achieved an accuracy of 0.971 and a macro-F1 score of 0.950, with identical reported metrics under plaintext and ciphertext-based inference. smash2026research papersprivacy](/r/smash2026/private-misinfo-inference) [Towards Transparent Reward Design in Clinical Reinforcement Learning 4-8 mins Using automated insulin dosing as a testbed, we show that conventionally trained policies occupy one such undisclosed point on a tradeoff surface, producing clinically consequential failure modes invisible to standard metrics. We construct an approximate Pareto frontier through preference-conditioned RL and show that it recovers policies with similar aggregate performance but different risk profiles across three simulated patients. smash2026research papersclinical](/r/smash2026/transparent-reward) [Tracing Bias Propagation Across Subgroups in Federated Learning 2-5 mins We show that the impact of false-negative errors on recipient client–subgroup pairs can be predicted from the global model's curvature together with source and recipient-group feature summaries, without retraining a model for each possible error. Notably, a non-target subgroup can experience the largest degradation. smash2026research papersprivacy](/r/smash2026/fl-bias-propagation) [NeuroBuilder: Auditable Agentic Data Curation for Neuro-Foundation Models 2-4 mins We present NeuroBuilder, an agentic adaptation workflow that combines semantic reasoning with deterministic extraction, validation, provenance, and execution controls while reserving consequential decisions for researchers. TUEG received a full pass with zero fabricated events. smash2026research papersllm-evaluation](/r/smash2026/neurobuilder) [From Extraction Accuracy to Clinical Usefulness: A Human-Centered Evaluation Framework for Source-Traceable Patient Summaries 3-6 mins A fact can be extracted correctly and still mislead if it is presented without temporal context, detached from its source, merged with conflicting information, or rewritten in a way that changes perceived urgency, certainty, or relevance. The framework combines deterministic checks, review of clinical meaning, and task-based evaluation of how users interpret and verify summaries … smash2026research papersllm-evaluation](/r/smash2026/source-traceable-summaries) [When Engagement Is Not Enough: What Should AI Optimize in Gamified Education? 2-4 mins In crossover STEAM education (n = 49), interest favored gamified instruction (dz = 0.64), whereas perceived expression acquisition favored conventional instruction (dz = -0.55). In disaster education (n = 37), engagement was high and perceived knowledge and competence increased, but intention changed little. smash2026research paperspeople](/r/smash2026/engagement-not-enough) [Towards Robust, Performance-Based Contamination Detection for LLMs 2-4 mins Our evasion is an argmax-preserving logit transform that makes signals on contaminated items resemble those on unseen items, while leaving every benchmark answer unchanged. We then introduce a detector based on Item Response Theory (IRT) residuals, which calibrates item difficulty on a population of 102 LLMs and flags items where a model outperforms its estimated ability. Because it reads only binary responses, its AUROC is exactly unchanged under the evasion. smash2026research papersllm-evaluation](/r/smash2026/irt-contamination) [Two for One: Privileged Bilateral Learning for Monocular Retinal Disease Classification 2-4 mins We thus propose a privileged learning framework in which a teacher model has access to bilateral CFP images during training and learns representations informed by information from both eyes, while the student is trained to transfer this knowledge from the bilateral teacher using only a single-eye image as input. At inference, the student operates exclusively on this single-eye input and does not require the contralateral image. smash2026research papersclinical](/r/smash2026/privileged-bilateral) [Damned if You Do(n't): How Epistemic Technology Spoils Expertise 3-6 mins We advance a concept of spoiled expertise which takes distinct forms based on two dimensions: knowledge workers' stance of adoption of epistemic technology, and locus of evaluation of expertise. Together these two dimensions culminate into four forms of spoiled expertise: (1) slopped expertise, (2) lagging expertise, (3) shunned expertise, and (4) outdated expertise. smash2026research paperspeople](/r/smash2026/spoiled-expertise) --- Source: https://jetty.io/docs/integrations/bring-your-own-agent # Bring Your Agent Bringing your own agent? Guide me through installing https://www.jetty.io/agent-instructions.md and setting up my first runbook. [Full instructions](/docs/integrations/bring-your-own-agent#install-the-plugin) ## Machine Instructions 🐦 **Onboarding an agent, not a human?** Point it at [jetty.io/agent-instructions.md](/agent-instructions.md) — a pasteable, numbered trigger that ends in a report-back block. The deep reference lives at [jetty.io/MACHINE\_CONTEXT.md](/MACHINE_CONTEXT.md). ## Get a token Sign up at [jetty.io](https://jetty.io), create a key under [Settings → API keys](/settings/api-keys) (it looks like `mlc_…`), and export it so the CLI and SDK can find it: Shell ``` export JETTY_API_TOKEN="mlc_…" ``` If your agent has the plugin installed, skip the dashboard: the `jetty-setup` skill below writes the token to `~/.config/jetty/token` for you. ## Install the plugin The plugin is the fastest way to use Jetty from an agent. It bundles four skills and the [MCP server](/docs/integrations/mcp-server) together, so a single install gives you natural-language commands, a guided setup wizard, and the runbook-authoring loop without leaving your editor. The skills are plain markdown, so any agent that supports skills can load them: the repo carries plugin manifests for Claude Code, Codex, and OpenCode, and Antigravity loads them once they are copied into its skills directory. In Claude Code, two commands add the marketplace and install the plugin: Shell ``` claude plugin marketplace add jettyio/jettyio-skills claude plugin install jetty@jetty ``` Then run the setup wizard once. It creates or links your account, writes your token to `~/.config/jetty/token`, and checks that your model provider keys are in place. In Claude Code that is a slash command; in other agents, ask to “set up Jetty” and the agent picks the skill from its description: Shell ``` /jetty-setup ``` That's it. The token it writes is the same one every other [on-ramp](/docs/get-started/connect-your-app) reads, so the MCP server, the SDK, and the CLI are all configured at the same time. Using an editor that speaks MCP but not skills, such as Cursor, VS Code, Windsurf, or Zed? Add the [MCP server](/docs/integrations/mcp-server) on its own and point the agent at [agent-instructions.md](/agent-instructions.md) for onboarding. The tools are identical; you drive the create and optimize loops in plain English instead of through a skill. ## Jetty Skills Each skill is a markdown file the agent reads on demand. In Claude Code they are slash commands; elsewhere, describe what you want and the agent matches the skill by its description. ### `jetty`: natural-language workflow control Talk to Jetty in plain English. The skill maps your request onto the underlying API, so you don't have to remember endpoints or JSON shapes. Text ``` /jetty list collections /jetty list tasks in acme /jetty run triage in acme with ticket="customer can't log in" /jetty show the last run for triage ``` ### `jetty-setup`: first-run onboarding The wizard you run once. Handles account creation, token storage, and a provider-key pre-flight so your first runbook doesn't fail on a missing key. ### `create-runbook`: author a new runbook Guided authoring for a brand-new runbook: it asks what you want done, what “done” looks like, and what to check, then writes a structured `RUNBOOK.md` with frontmatter and evals. See [Runbooks](/docs/ai-workloads/runbooks) for the anatomy. ### `optimize-runbook` : Improving an existing runbook It reads your prior [runs](/docs/ai-workloads/runs), finds where they fall short of the runbook's evals, proposes targeted edits, and re-runs to confirm the score moved. The plugin ships the same [MCP server](/docs/integrations/mcp-server) the rest of the editors use, so when you ask for a run through the `jetty` skill, your agent is calling the `run-workflow` MCP tool against `flows-api.jetty.io`. The skills are the guided layer on top; the MCP tools work in every editor. --- Source: https://jetty.io/docs/get-started/connect-your-app # Connecting Your App Jetty is one API with swappable parts. Everything below talks to the same backend. We support swappable AI gateways (and models), agent runtimes, and frameworks. Use through our Client SDK or call Jetty's API directly. [AI Gateways](/docs/ai-workloads/runtime-configuration#ai-gateway) [Agent Runtimes](/docs/ai-workloads/runtime-configuration#agent-runtimes) [Client SDK ![Pelly, the Jetty pelican](/_next/image?url=%2Fpelly-pixel.png&w=128&q=75)](/docs/integrations/sdk) [Agent Frameworks](/docs/jetty-and-friends/bring-your-friends-and-framework) ## Connect in three steps 1. **Get a token.** Create a key under [Settings → API keys](/settings/api-keys). The Client SDK, the MCP server and the API all read the same token. 2. **Pick an on-ramp.** Install [`@jetty/sdk`](/docs/integrations/sdk) in your app, add the [MCP server](/docs/integrations/mcp-server) to your editor, or call the [REST API](/docs/reference/api) directly with a bearer token. 3. **Choose the parts.** Pick an [AI gateway and an agent runtime](/docs/ai-workloads/runtime-configuration) for the workload. Both are runtime configuration, not app code, so you can swap them per run. ## Open source projects - [jettyio/jettyio-skills](https://github.com/jettyio/jettyio-skills): the Claude Code plugin, the MCP server, and the shell CLI. - [jettyio/jetty-sdk](https://github.com/jettyio/jetty-sdk): `@jetty/sdk`, the TypeScript client, plus `@jetty/eve` and the eve and Flue worked examples. --- Source: https://jetty.io/docs/product-overview # Product Overview Learn how to measure and manage AI workloads. Jetty provides the key components required to be able to author, run, investigate and improve your AI workloads. ## What is Jetty? Jetty handles AI workloads using plain language instructions, called [runbooks](/docs/ai-workloads/runbooks). These AI workloads run in a virtual machine called a sandbox, and are paired with a coding agent, a model and all the integrations you need to get going. You provide an objective, define evaluation criteria and Jetty puts together the steps to produce the outcome. These jobs can run for several minutes or longer and their outputs can be evaluated and shared from the web interface. Each execution of a Jetty workload includes the following elements. [Running AI WorkloadsHow a runbook becomes a running workflow: the author → run → investigate → improve lifecycle, and what Jetty assembles around every run.](/docs/product-overview/running-ai-workloads)[Investigating Agent QualityHow to read a run and its verdict, add a check at the failure moment, compare runs honestly, and run the investigate-and-improve loop.](/docs/product-overview/investigating-agent-quality) --- Source: https://jetty.io/docs/product-overview/running-ai-workloads # Running AI Workloads This page describes the [emergent patterns in how AI workloads function](https://blog.jetty.io/p/patterns-were-the-map-in-the-search) and the best way to author, run, investigate and improve their quality. ## Can Jetty help me solve this task? An AI workload is an automation-ready task that requires human judgement and can be meaningfully measured by an expert. Before delegating the job to Jetty, ask yourself: can you define what a good result looks like, and will the task come up again? If the answer is yes to both, then this is a good candidate for running it on Jetty. Repeatable work with no real quality bar is basic automation. You may want to benchmark a basic automation with deterministic scripts against an AI workload. For instance, processing hundreds of documents is a good example of an AI workload, whereas a back-and-forth with a chatbot might be best done with another tool. Jetty runs plain language instructions, called runbooks, in a virtual machine called a sandbox with a coding agent. ## Runbooks: the agent workflow description A Jetty [runbook](/docs/ai-workloads/runbooks) is a markdown file that provides an AI agent all the information required to perform a long-running task to a pre-defined standard of quality. While it might be tempting to think of runbooks as just fancy prompts, there's a little more going on under the hood. If you haven't installed Jetty yet, start with [Bring your own agent](/docs/integrations/bring-your-own-agent): the plugin, the `jetty-setup` skill, and your first run. The same deployed runbook can be re-run with a different model, agent, or input without editing the file, which is what makes model and agent sweeps a loop instead of a project. The exact workflow shape and the parameter-resolution rules are in the [machine instructions](/docs/reference/api/machine-instructions). ## Runtime configuration The runtime configuration characterizes which model and agent invoke the runbook, the environment they are given, and the tools they have access to. Each runtime's provider compatibility, model id format, and a configuration that runs are in [Runtime Configuration](/docs/ai-workloads/runtime-configuration). ### Bring Your Own AI Key (Optional) Runbooks can also use an AI Gateway of your choosing: name a model and a `model_provider` and Jetty routes accordingly. By default, we use an OpenRouter multi-provider gateway. Learn more about [setting up your own AI Gateway](/docs/ai-workloads/runtime-configuration#ai-gateways). Provider keys are configured for your whole account (Settings → Bring Your Own Key) and injected into the sandbox at run time. The same endpoint doubles as a plain LLM proxy with a chat-completions call without a runbook attached is passthrough mode, routed across 100+ providers, with every call recorded as a run. ### Tools, skills, and MCPs The agent runtime brings its own tools as part of the runbook. This includes file editing, shell, web navigation etc. You choose it with the `agent:` frontmatter field of the runbook. Runbook tasks can also attach MCP servers. These can be declared as either a local command or a remote HTTP endpoint. See more details in the [machine instructions](/docs/reference/api/machine-instructions). Skills go inside runbooks since they are already plain markdown. ## Integrations Your AI Workload will probably need to connect to some system or push data into something downstream. Fortunately, agents are clever enough, given the right integration details, to connect appropriately. Each workload manages its own integrations in the form of environment variables defined as part of the runbook. Learn more about [setting up integrations and credentials](/docs/integrations). --- Source: https://jetty.io/docs/product-overview/investigating-agent-quality # Investigating Agent Quality When an AI workload completes successfully, you know that the runtime configuration was correct, but not whether the output is at a level of quality required. This page discusses how we can investigate agent quality using evaluation tools. Look at [agent evaluations](/docs/ai-workloads/agent-evaluations) to learn more about which tools are available. One thing you will notice is that evaluation is not a one-tool-fits-all sort of thing. We discuss it [how you need pass/fail and directional tests](https://blog.jetty.io/p/tests-are-boolean-evals-are-a-gradient) for AI workloads to really succeed. Investigating agent quality starts with looking at how individual runs are performing. Since Jetty captures all the inputs and outputs, it's easy to use Jetty to investigate how a single run went and then how to improve it. The same strategy can be applied to some common use cases: - Surfacing AI costs - Identifying LLM errors - Picking the best model for the job Once you are getting results, you can share them with your team or [schedule the workload](/docs/guides/scheduling-ai-workloads) to run regularly. ## Start with the run Every execution is captured as a [run](/docs/ai-workloads/runs): a complete, replayable trace of each step, the inputs it received, the outputs it produced, the files it wrote, and how long it took. When a result looks wrong, you don't guess at what the agent did; you open the run and read it. The log shows which tools and skills the agent reached for, and the step boundaries show where the work went sideways. ## Investigating the results Use the Jetty agent skill to pull down the runs generated from an AI workload. From there you can launch an investigation, read the agent logs and see how the agent performed the task. ### Identify LLM errors Most agent harnesses are resilient to failure; if something doesn't work the first time, they will try and find an alternate route. For instance, if accessing the web through a terminal command fails, they may spin up a web browser and navigate to the page instead. Jetty logs every step and those misteps can be used to [improve the agent runbook](/docs/guides/improving-agent-runbooks). ### Surface costs The bulk of AI spend often comes from the cost for tokens. AI gateways and model provider costs vary wildly and they are often time a proxy for the reasoning performance of a model. Jetty makes it easy to take an AI workload and measure it across multiple models. The evaluations (code checks, checklists and judges) provide the quality dimension. Cost, time to completion and token consumption provide the other half of the equation. ### Picking the best model for the AI workload With investigations suited to your task, you can now make an informed decision and share the results with your team. Sorting the cases by judge-scored complexity shows where each model's success rate falls off. The cheapest model whose reach covers your cases is usually the right one. ## Add a check at the failure moment Most evals get written after a run disappoints, not before, and that's fine. When a run fails in a way nothing caught, use the agent skill to modify the runbook with an evaluation: a code check if you can assert it, a checklist item if the agent stopped early, a judge criterion if it takes a reader. Before you rely on a new code check, run it against the outputs of a few recent passing runs. A failed check fails the run's verdict, so a buggy check fails the verdict of every run and takes your trust in the whole eval with it. If the passing runs fail your new check, the check is wrong, not the runs. Budget the instruments by cost. Code checks and the checklist are free to run every time, in flight. Judges cost tokens: score every run with a cheap LLM judge if you like, and send a sample to an agent judge for the verdict you trust. ## Investigate and improve Investigation only matters if it changes the next run. Because every run is recorded in full and every runbook carries its evals, the loop is short: 1. **Run** the runbook over a few cases. Each run records a verdict. 2. **Read** the runs that failed. The trace shows which step fell short and which check caught it. 3. **Edit** the runbook: add the missing check, tighten a step, fix a threshold. 4. **Re-run** and confirm the verdicts improved. Repeat. `optimize-runbook` is a skill in the [Jetty plugin](/docs/integrations/bring-your-own-agent) that runs this loop for you. Point it at a runbook and it reads the prior runs, finds the failure patterns, proposes targeted edits (surgical changes, not a rewrite), and re-runs the evals to confirm the change actually helped before you keep it. No plugin? Any MCP-connected agent can do the same by hand with the [MCP server](/docs/integrations/mcp-server): `list-trajectories` and `get-trajectory` to read the runs, `update-task` to apply the edit, and `run-workflow` to re-check. ### Keep scores honest A rising average is not proof the workload got better; it might mean the judge got easier to please. Read the trend across iterations rather than a single number. And keep checking the judge against a person: when you disagree with a verdict, label the run with your own, and treat agreement between your labels and the judge's as the health metric for the eval itself. If that agreement drifts, fix the judge before you trust another score from it. --- Source: https://jetty.io/docs/ai-workloads # AI Workloads An AI workload is an automation-ready task that requires human judgement and can be meaningfully measured by an expert. If you can define what a good result looks like, and the task will come up again, it's a workload Jetty can run. See [Running AI Workloads](/docs/product-overview/running-ai-workloads) for the full lifecycle. [RunbooksThe hero artifact: one markdown file with the job, the bar for “done”, and the checks.](/docs/ai-workloads/runbooks)[Runtime ConfigurationThe agents Jetty runs inside its own sandboxes, which providers back each, and how to pick one.](/docs/ai-workloads/runtime-configuration)[SandboxesThe isolated machine each run gets: what the default sandbox contains, its lifecycle and limits, where files go, and how to run on your own Docker image.](/docs/ai-workloads/sandboxes)[Secrets and Environment VariablesWhere a secret is stored, how it travels from your workspace into the sandbox, which process receives it, and what Jetty keeps out of the recorded run.](/docs/ai-workloads/secrets-and-environment-variables)[RunsEvery run is captured end to end: each step, its inputs, outputs, files, and status, plus labels to group and compare runs.](/docs/ai-workloads/runs)[Agent EvaluationsAgent checks, Jetty checks, checklists and judges live in the runbook and run in flight. How the report comes together, the verdict, strict evaluation, and a failed run compared with a failed evaluation.](/docs/ai-workloads/agent-evaluations)[REST APIThe Jetty REST API: run runbooks as tasks (deploy, run, monitor — the primary endpoint), the OpenAI-compatible chat-completions compatibility layer, schedules, webhooks, and GitHub-PR APIs.](/docs/reference/api) --- Source: https://jetty.io/docs/ai-workloads/runbooks # Runbook.md: Markdown Job Specs for AI Agents A runbook is a job spec for an AI agent, written as one markdown file (conventionally `RUNBOOK.md`). It says what the agent should do, what “done” looks like, and the checks that prove it. Because it is plain markdown, a runbook is readable by people, forkable like any file, and portable across models and agents; Jetty runs it in a sandbox and records every run. There are three ways to get started: 1. **[Ask Pelly](/flows)**. Let Pelly, the Jetty agent, guide you through the creation process in minutes. 2. **[Bring your agent.](/docs/integrations/bring-your-own-agent)** Use the Jetty Skill and MCP to bring your agent context into runbook creation. 3. **[Try a Project.](/runbooks)** Fork a public runbook from the directory. ## What a runbook looks like The markdown file has four parts: frontmatter, setup, steps, and evaluations. The frontmatter is a YAML block at the top that declares a default runtime, model and agent configuration. Note that these can be overridden when the AI workload runs with a specified [runtime configuration](/docs/ai-workloads/runtime-configuration). The setup section is a short description of the environment the work needs. The steps section describes the sequence of instructions the agent should follow to complete the job. Finally, the evaluations section describes the criteria for success and which files the agent should produce. The frontmatter fields are covered in [runtime configuration](/docs/ai-workloads/runtime-configuration), the evaluation styles in [agent evaluations](/docs/ai-workloads/agent-evaluations), and parameters and uploaded files under Runbook context below. ## Writing runbooks You can write a runbook in two places. In the Jetty web application, Pelly guides you through it: describe the job, and it drafts the runbook with its frontmatter, steps and evaluations in place, ready to run. With the Jetty agent CLI, your own coding agent does the same from your editor using the `create-runbook` skill, so the file lives next to your code and under version control. Either way you end up with a `RUNBOOK.md` you can read, run and fork. Once it runs, the work shifts to modifying and improving it from the evidence its runs produce. That loop, pulling recent runs, tightening the evaluations they expose and editing the file, is covered in [Improving agent runbooks](/docs/guides/improving-agent-runbooks). ## Why runbooks are different from prompts and skills Runbooks are borne out of the lived experience of [watching AI workloads fail in unpredictable ways](https://blog.jetty.io/p/runbooks-what-agents-need-to-hill). That structure is what separates a runbook from the other ways of telling an agent what to do: | | Agent prompt | Agent skill | Runbook | | --- | --- | --- | --- | | **What it captures** | A one-off ask | How to do something, reusably | A whole job: setup, task, checks, outputs | | **Definition of done** | Whatever the agent decides | Inherited from whoever invokes it | Explicit criteria in the file | | **Checking the work** | You read the output | None built in | The agent scores its output and iterates until it passes | | **Output** | A reply in a session | Depends on the session | Declared artifact files, persisted on every run | | **Repeatability** | Copy, paste, hope | Consistent method, unmanaged results | Parameterized, deployable, schedulable, comparable run to run | Each execution produces a [run](/docs/ai-workloads/runs), which includes the inputs, outputs, logs, telemetry, and a copy of the runbook. ## Working with runbooks Runbooks serve two purposes inside Jetty: executing a long-running AI workload, and running investigations across runs to figure out how your AI workloads are performing. Get started on the Jetty web platform, then keep managing them by [bringing your own agent](/docs/integrations/bring-your-own-agent) to Jetty. | Activity | [Jetty Web](/flows) | [Jetty Agent CLI](/docs/integrations/bring-your-own-agent) | | --- | --- | --- | | **Author** | [Building a runbook](#writing-runbooks) | Yes | Yes | | Runbook editing | Beta | Yes | | **Run** | Launch runs | Yes | Yes | | [Schedule workloads](/docs/guides/scheduling-ai-workloads) | Beta | Yes | | **Investigate** | Model benchmarking | Beta | Yes | | Cost and quality analysis | Beta | Yes | | **Improve** | [Add evaluations](/docs/ai-workloads/agent-evaluations) | Beta | Yes | ## Runbook Context The runbook markdown itself is the agent's instruction. Three mechanisms parameterize it: - **Template variables.** `{{name}}` placeholders substitute from the run's `vars`, declared with their defaults in the runbook's `## Parameters` table. A run's `vars` fill the table key by key: a run that sets one input keeps every other input's default. A Default of `*(required)*` marks an input with no default; a run that leaves it blank fails before the sandbox starts, naming the input. `{{results_dir}}`, `{{checks_dir}}` and `{{assets_dir}}` are the sandbox's own paths and cannot be set by a run. Substitution is literal string replacement — declare every variable, and restate parameters imperatively in the body; agents ignore values that only appear in metadata. - **Uploaded files.** Files attached to a run are mounted in the sandbox and their listing is appended to the instruction automatically. - **Prior runs.** In multi-step workflows, [path expressions](/docs/reference/api/steps#path-expressions) pull the outputs of earlier [actions](/docs/reference/api/steps) into the instruction. One hard limit: the instruction after substitution must fit in ~120 KB (the instruction is passed to the agent CLI as a single argument). Large corpora belong in uploaded files, not inlined variables. How files get into and out of the sandbox, and how the agent reaches the web, is covered under [MCPs & Skills](/docs/ai-workloads/runtime-configuration#mcps-and-skills) in Runtime Configuration. --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration # Runtime Configuration You don't need to set any of this to get started. Every run comes preset with sensible defaults that work for most workloads, so you can focus on the task, not the infrastructure. Change them when a workload calls for it. When Jetty runs a [runbook](/docs/ai-workloads/runbooks), it executes a coding agent inside a managed sandbox. You choose _which_ agent and which provider backs it. The following configuration options will determine how the runbook is executed. - The [**AI Gateway**](#ai-gateway) determines which model provider will generate the tokens required. Each provider has their own addressing scheme for models. Jetty uses an OpenRouter compatible gateway by default. - The [**Sandbox**](#sandboxes) is the isolated machine the agent runs in. The default comes with common software packages and libraries (e.g. Python and Node) and a web browser. - [**Agent Runtimes**](#agent-runtimes): sometimes called the agent harness, is the command-line tool that runs the runbook. The model takes input in the form of text and images and the agent provides the memory, and loop orchestration. - [**Model**](#models). This is the foundation model used by the agent. You can try different models with the same runbook and evaluate performance. - [**MCPs & skills**](#mcps-and-skills) are additional plugins that give the agent specialized capabilities: web browsing, spreadsheet manipulation, access to Google Workspace, etc. ## AI Gateways AI providers (or model providers) are the backends Jetty calls for inference, both from the [passthrough proxy](/docs/reference/api) and from the model steps inside an AI workload. Each needs a key in the workspace; configure them in [Settings → Bring Your Own Key](/settings/ai-providers). [Anthropic Claude models via `anthropic/…` slugs, with a direct API key.](https://www.anthropic.com/claude) [AWS Bedrock Models hosted on Bedrock via `bedrock/…` slugs, with bearer-token auth.](https://aws.amazon.com/bedrock/) [Cloudflare AI Gateway Anthropic and OpenAI models through your own gateway, via the same `anthropic/…` and `openai/…` slugs as OpenRouter.](https://developers.cloudflare.com/ai-gateway/) [Google Gemini Gemini models via `gemini/…` slugs; Vertex too.](https://ai.google.dev/gemini-api/docs/models) [OpenAI GPT and o-series models via `openai/…` slugs.](https://platform.openai.com/docs/models) [OpenRouter A gateway to hundreds of upstreams via `openrouter/…` slugs, and the default provider.](https://openrouter.ai/models) [vLLM Your own self-hosted, OpenAI-compatible endpoint.](https://docs.vllm.ai/) What each `model_provider` needs in the workspace: | `model_provider` | Required | Optional | | --- | --- | --- | | `anthropic` | `ANTHROPIC_API_KEY` | — | | `openai` | `OPENAI_API_KEY` | — | | `bedrock` | `AWS_BEARER_TOKEN_BEDROCK` | `AWS_REGION` | | `openrouter` | `OPENROUTER_API_KEY` | `OPENROUTER_BASE_URL`, `OPENROUTER_HTTP_REFERER`, `OPENROUTER_X_TITLE` | | `cloudflare` | `CLOUDFLARE_AI_GATEWAY_ACCOUNT_ID`, `CLOUDFLARE_AI_GATEWAY_TOKEN` | `CLOUDFLARE_AI_GATEWAY_ID` (default `default`), `CLOUDFLARE_AI_GATEWAY_BASE_URL` | | `google` | `GEMINI_API_KEY` | `GOOGLE_API_KEY` | | `vllm` | `VLLM_BASE_URL` | `VLLM_API_KEY` | `cloudflare` is your own [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/) in front of the provider-native Anthropic and OpenAI endpoints, so each runtime speaks the wire it would speak on a direct key. The gateway must be authenticated: Jetty sends only the gateway token, never your Anthropic or OpenAI keys, and the upstream is paid for either by provider keys you store in Cloudflare (BYOK) or by Unified Billing credits. Model ids are vendor-namespaced exactly as on OpenRouter (`anthropic/claude-sonnet-5`, `openai/gpt-5.5`), so moving a runbook between the two gateways is a one-line change of `model_provider`. Prompts and completions are logged in your Cloudflare account when the gateway's logging is on. `antigravity` does not support it, and Google models are not served through it. `vllm` is the catch-all for a self-hosted or custom endpoint — vLLM, SGLang, TGI, LiteLLM, or any OpenAI- or Anthropic-compatible server. Which wire it must speak depends on the runtime, as noted on each runtime's page. Not every model needs a card of its own, either: the passthrough proxy and the `litellm_chat` / `litellm_batch` [actions](/docs/reference/api/steps) route to 100+ providers through LiteLLM, and the action catalog adds image and video models. See [Integrations](/docs/integrations) for how keys reach the sandbox. ## Sandboxes Every run gets its own sandbox, created for the run and deleted when it ends. - **Default environment:** the `python312-uv` snapshot, with Python 3.12, `uv`, Node 22, Chromium and Playwright, and the claude-code, codex, opencode and hermes CLIs pre-installed. Other runtimes install when the run starts, which costs a few seconds. - **Default size and time:** 4 CPUs, 8 GB of memory, 10 GB of disk, and up to 30 minutes for the agent. For longer jobs, contact [support@jetty.io](mailto:support@jetty.io). - **Files:** the agent works in `/app`, reads attached files from `/app/assets` and must write deliverables to `/app/results`. - **Your own environment:** choose another snapshot with `snapshot:` in frontmatter, or run on a public Docker image with the `image` run parameter. [Sandboxes](/docs/ai-workloads/sandboxes) covers the lifecycle, the directory layout, limits, network access and running on a custom Docker image. ## Agent Runtimes Each runtime executes inside the sandbox, reads the matching provider key, and is chosen with an `agent:` line in runbook frontmatter or `agent` in a run's `init_params`, both covered below. [antigravity Google's Antigravity CLI (`agy`).](/docs/ai-workloads/runtime-configuration/antigravity) [claude-code Anthropic's coding agent, and the default.](/docs/ai-workloads/runtime-configuration/claude-code) [codex OpenAI's coding agent.](/docs/ai-workloads/runtime-configuration/codex) [opencode The open-source SST agent.](/docs/ai-workloads/runtime-configuration/opencode) [pi The pi coding agent. No MCP client, by design.](/docs/ai-workloads/runtime-configuration/pi) The **Providers** column lists the values you can put in `model_provider` for that runtime; **MCP** says whether the runtime honours `mcp_servers` on a task; **Usage** says whether token counts land on the run. | Runtime | Providers | MCP | Usage | Use it when | | --- | --- | --- | --- | --- | | [`claude-code`](/docs/ai-workloads/runtime-configuration/claude-code) (default) | `anthropic`, `bedrock`, `openrouter`, `cloudflare` (Anthropic models), `vllm` | ✓ | ✓ | The starting point. Best general agentic coding behaviour; the loop the skills are tuned against. | | [`codex`](/docs/ai-workloads/runtime-configuration/codex) | `openai`, `openrouter`, `cloudflare` (OpenAI models), `vllm` | — | ✓ | You want OpenAI models driving the agent. | | [`antigravity`](/docs/ai-workloads/runtime-configuration/antigravity) (Google) | `google` | ✓ | ✓ | You want Gemini models driving the agent. Google's current terminal agent, `agy`. | | [`opencode`](/docs/ai-workloads/runtime-configuration/opencode) | `anthropic`, `openai`, `bedrock`, `openrouter`, `cloudflare`, `vllm` | ✓ | best-effort | You want an open-source agent with MCP, browser automation and scraping ready on every run. | | [`hermes`](/docs/ai-workloads/runtime-configuration/hermes) (Nous) | `anthropic`, `openai`, `bedrock`, `openrouter`, `cloudflare`, `google`, `vllm` | — | ✓ | You want one runtime across every provider, including a self-hosted vLLM endpoint. | | [`pi`](/docs/ai-workloads/runtime-configuration/pi) | `anthropic`, `openai`, `bedrock`, `openrouter`, `cloudflare`, `google`, `vllm` | — | ✓ | You want a lean agent that leans on CLI tools and skills rather than MCP. | _Best-effort_ usage means the runtime doesn't emit token counts reliably, so a `null` or zero `usage` on the run is not a failure signal. New to this? Stay on `claude-code` until you have a reason to switch. ## Models The model is the foundation model the agent calls: text and images in, text out. Name it with `model` in the provider's own format, paired with the `model_provider` that serves it; each runtime's page lists the formats it accepts. Because the model is a setting on the run rather than part of the runbook, you can run the same runbook on several models and compare the runs. Below are some models Jetty users run frequently. | Model | `model` | `model_provider` | Tier | | --- | --- | --- | --- | | Claude Opus 5 | `anthropic/claude-opus-5` | `anthropic`, `bedrock`, `openrouter`, `cloudflare` | Frontier | | GPT-5.4 | `openai/gpt-5.4` | `openai`, `openrouter`, `cloudflare` | Frontier | | GPT-5.6 Sol | `openai/gpt-5.6-sol` | `openai`, `openrouter`, `cloudflare` | Frontier | | Kimi K3 | `moonshotai/kimi-k3` | `openrouter` | Frontier | | Claude Sonnet 5 (default) | `anthropic/claude-sonnet-5` | `anthropic`, `bedrock`, `openrouter`, `cloudflare` | Balanced | | Claude Sonnet 4.6 | `anthropic/claude-sonnet-4.6` | `anthropic`, `bedrock`, `openrouter`, `cloudflare` | Balanced | | GPT-5.6 Terra | `openai/gpt-5.6-terra` | `openai`, `openrouter`, `cloudflare` | Balanced | | Gemini 3.7 Flash | `google/gemini-3.7-flash` | `google`, `openrouter` | Balanced | | GLM 5.2 | `z-ai/glm-5.2` | `openrouter` | Balanced | | GPT-5.6 Luna | `openai/gpt-5.6-luna` | `openai`, `openrouter`, `cloudflare` | Fast | | GLM 5.3 Flash | `z-ai/glm-5.3-flash` | `openrouter` | Fast | | DeepSeek V4 Flash | `deepseek/deepseek-v4-flash` | `openrouter` | Fast | A model's **Tier** is a rough positioning, in terms of price and speed. Generally faster models are cheaper. Opus-class models are blocked on the free trial, so they need your own key. Anything else in the [OpenRouter model directory](https://openrouter.ai/models) works too, with `model_provider: openrouter`. There are three places to set the model. ### Runbook Frontmatter sets the default A runbook names its runtime in frontmatter, so the choice travels with the file: YAML ``` --- agent: claude-code # claude-code | codex | antigravity | opencode | hermes | pi model: anthropic/claude-sonnet-5 # slug in the provider's own format model_provider: openrouter # anthropic | openai | bedrock | openrouter | cloudflare | google | vllm snapshot: python312-uv # the default sandbox notify_on_completion: true # start "Email me when this run finishes" ticked --- ``` `notify_on_completion` only seeds that checkbox on the dashboard's launch forms; whoever starts the run can still flip it, and runs started from the API or a routine do not read it. Leave all of them out and you get `claude-code` on `anthropic/claude-sonnet-5` via `openrouter` in the `python312-uv` sandbox. Set `model` without `model_provider` and Jetty picks the provider: OpenRouter if the workspace has an `OPENROUTER_API_KEY` and the runtime supports it (`codex` is excluded; see [its page](/docs/ai-workloads/runtime-configuration/codex)), then Cloudflare AI Gateway if there is a `CLOUDFLARE_AI_GATEWAY_TOKEN` and the runtime supports it, then Bedrock if there is an `AWS_BEARER_TOKEN_BEDROCK`, then the runtime's native provider — Anthropic for `claude-code`, OpenAI for `codex`, Google for `antigravity`. `vllm` is never picked for you. `opencode`, `hermes` and `pi` have no native provider, so name `model_provider` for them. Naming it is the safe habit everywhere; the inference is a convenience that will eventually go away. ### Override it per run The frontmatter is a default, not a lock. When you deploy a runbook, Jetty stores `agent`, `model`, `model_provider` and `snapshot` as `init_params` on the task, and the run step reads them from there. Whatever you send when you start a run merges over those defaults, caller wins. So a deployed runbook can run on another agent, model or provider without a redeploy and without a second copy of the file. That is what turns a model sweep into a loop instead of a project. Two ways to send them. From the run endpoint, pass `init_params`: Shell ``` curl -X POST https://flows-api.jetty.io/api/v1/run/my-workspace/cbc-homepage-summary \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "init_params": { "agent": "antigravity", "model": "gemini-3.7-flash", "model_provider": "google", "agent_env": { "ANTIGRAVITY_EFFORT": "medium" }, "vars": { "url": "https://www.cbc.ca/" } } }' ``` The run form in the web app and the `run-workflow` tool in the [MCP server](/docs/integrations/mcp-server) send the same `init_params`, so anything below applies there too. ### The on-ramps Every on-ramp hits the same backend ( `https://flows-api.jetty.io` ) with the same token, and the API is OpenAI-compatible, so you can start with the plugin today and drop down to the SDK or REST API later without changing anything server-side. Each of them can send the same `init_params`. | On-ramp | Best for | You get | | --- | --- | --- | | [Bring your own agent](/docs/integrations/bring-your-own-agent) | Agents with skills (Claude Code, Codex, OpenCode) | The `jetty` skill for natural-language control, the `jetty-setup` wizard, and the `create-runbook` / `optimize-runbook` skills. | | [MCP server](/docs/integrations/mcp-server) | Cursor, VS Code, Windsurf, Zed, Codex | 16+ Jetty tools in any MCP-compatible agent, via `npx -y jetty-mcp-server`. | | [Client SDK](/docs/integrations/sdk) | TypeScript / Node apps & CI | `@jetty/sdk`, a typed `JettyClient` with `runAndWait()` and the full run / result / label surface. | | [REST API](/docs/reference/api) | Any language, any runtime | The tasks API (deploy, run, poll — the primary endpoint), plus an OpenAI-compatible `/v1/chat/completions` layer for compatibility. | | Standalone CLI | Shell scripts & quick checks | `source jetty-cli.sh` for `jetty_collections`, `jetty_run_sync`, and friends. No install. | ## MCPs & Skills Every runtime brings its own core tools: file editing, a shell, and web navigation. MCP servers and skills extend that set. A task can attach MCP servers, declared as a local command or a remote HTTP endpoint, for capabilities like browser automation, spreadsheet manipulation, or Google Workspace access; the **MCP** column in the runtime table shows which runtimes honour `mcp_servers`. Skills are plain markdown, so they go inside the runbook itself. The exact shapes are in the [machine instructions](/docs/reference/api/machine-instructions). Two capabilities every run relies on, whichever runtime you pick, are files and the web. ### Files and the web Files attached to a run are mounted at `/app/assets/`, and everything the agent writes to `/app/results/` is saved with the run; a run that ends with an empty results directory fails. Sandboxes have outbound network access, so agents can fetch URLs, call APIs, install packages and drive a browser. Details are in [Sandboxes](/docs/ai-workloads/sandboxes#layout). ## Alternative Integrations Prefer to keep the agent in your own stack? Check out how you can [bring your Friends and Framework](/docs/jetty-and-friends/bring-your-friends-and-framework) to Jetty. --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration/antigravity # antigravity Google's current terminal agent, `agy`. Its default sign-in is a browser flow, so Jetty runs it in Gemini-API-key mode instead: the sandbox gets a settings file with `modelProvider: gemini` and the workspace's `GEMINI_API_KEY`. Print mode, permissions auto-approved, a 120-minute print timeout in place of the CLI's five-minute default.
Providersgoogle
Model idsagy's own catalog, not Gemini API ids: gemini-3.7-flash, gemini-3.1-pro, each with an effort tier. A tiered slug (gemini-3.7-flash-medium) carries the tier itself. Default gemini-3.7-flash. gemini-2.5-pro is rejected.
MCPYes, through ~/.gemini/config/mcp_config.json; bearer tokens from mcp_auth_tokens are honoured.
UsageYes, including thinking tokens.
agent_envANTIGRAVITY_EFFORT (low / medium / high, default high; none omits the flag). ANTIGRAVITY_PRINT_TIMEOUT (a duration like 90m, default 120m).
The tier rules are agy's: a bare slug needs an effort, a tiered slug must not be combined with one, and `gemini-3.1-pro` only has `low` and `high`. Jetty normalises either form, so write whichever reads better and use `ANTIGRAVITY_EFFORT` to change the tier per run. `high` is the default because it is the one tier every catalog model accepts. YAML ``` agent: antigravity model: gemini-3.7-flash model_provider: google ``` Cheaper tier for a routine run, no file change: JSON ``` { "init_params": { "agent_env": { "ANTIGRAVITY_EFFORT": "medium" } } } ``` * * * Back: [Runtime Configuration](/docs/ai-workloads/runtime-configuration) — the full comparison table, frontmatter defaults, and [per-run overrides](/docs/ai-workloads/runtime-configuration#per-run). --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration/claude-code # claude-code Anthropic's agent, and the default [runtime](/docs/ai-workloads/runtime-configuration). Runs `claude -p` with a fixed tool allowlist (Bash, Edit, Write, Read, Glob, Grep, WebFetch, notebooks, todos, sub-agents) plus `mcp____*` for each MCP server you declare. Versions are pinned and bumped deliberately.
Providersanthropic, bedrock, openrouter, cloudflare (Anthropic models), vllm
Model idsBare on Anthropic and Bedrock (claude-sonnet-5, anthropic.claude-sonnet-5); vendor-namespaced on OpenRouter and, the same way, on Cloudflare AI Gateway (anthropic/claude-sonnet-5); verbatim on a custom endpoint.
MCPYes. Written to /app/.mcp.json; URL servers get a bearer header from mcp_auth_tokens.
UsageYes, including cache read and cache creation tokens.
agent_envANTHROPIC_AUTH_TOKEN for custom endpoints that only accept a bearer token; ANTHROPIC_CUSTOM_HEADERS to add request headers.
`vllm` here means any endpoint that speaks the Anthropic Messages API (a LiteLLM proxy, Bedrock Mantle's `/anthropic` path). The SDK appends `/v1/messages` itself, so `VLLM_BASE_URL` is the segment before that. On OpenRouter every request carries the run id as `x-session-id`, so traces group per run in OpenRouter's dashboard. YAML ``` agent: claude-code model: anthropic/claude-sonnet-5 model_provider: openrouter ``` * * * Back: [Runtime Configuration](/docs/ai-workloads/runtime-configuration) — the full comparison table, frontmatter defaults, and [per-run overrides](/docs/ai-workloads/runtime-configuration#per-run). --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration/codex # codex OpenAI's agent, pinned to a known-good release (`@latest` broke three times in one week). Runs `codex exec --json` with approvals and its own sandbox turned off, since the Jetty sandbox already isolates the run.
Providersopenai, openrouter, cloudflare (OpenAI models), vllm
Model idsBare on OpenAI (gpt-5.5); vendor-namespaced on OpenRouter and, the same way, on Cloudflare AI Gateway (openai/gpt-5.5); verbatim on a custom endpoint.
MCPNo. Declared servers are logged as a warning and ignored.
UsageYes.
agent_envOPENAI_REASONING_EFFORT (minimal to xhigh, model-dependent). OPENAI_WIRE_API only matters on older pins.
Two things trip people up. The Rust CLI ignores `OPENAI_BASE_URL`, so routing is written to `config.toml` and codex is left out of the OpenRouter auto-default: to run it on OpenRouter, say so. And codex 0.146 and later speak only the Responses API, so a chat-completions-only endpoint (plain vLLM) can't be driven by codex; use [`pi`](/docs/ai-workloads/runtime-configuration/pi), [`opencode`](/docs/ai-workloads/runtime-configuration/opencode) or [`hermes`](/docs/ai-workloads/runtime-configuration/hermes) for those. YAML ``` agent: codex model: gpt-5.5 model_provider: openai ``` The same runbook on OpenRouter, with more thinking, from the run endpoint: JSON ``` { "init_params": { "model": "openai/gpt-5.5", "model_provider": "openrouter", "agent_env": { "OPENAI_REASONING_EFFORT": "high" } } } ``` * * * Back: [Runtime Configuration](/docs/ai-workloads/runtime-configuration) — the full comparison table, frontmatter defaults, and [per-run overrides](/docs/ai-workloads/runtime-configuration#per-run). --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration/hermes # hermes Nous Research's agent, run as `hermes chat -Q`. The one runtime that reaches all seven providers from a single config, including a self-hosted vLLM endpoint. Jetty pre-seeds Hermes's provider file with its own entries so requests go where you pointed them rather than through Hermes's built-in aliases.
Providersanthropic, openai, bedrock, openrouter, cloudflare, google, vllm. Through Cloudflare AI Gateway it drives both vendors, but Hermes sends custom request headers on its OpenAI client only, so anthropic/* runs authenticate through the gateway without cf-aig-metadata attribution in its log.
Model idsVendor-namespaced on OpenRouter and Cloudflare AI Gateway (anthropic/claude-sonnet-5 on either), bare on native providers, and passed through untouched on vllm so HF-style ids like meta-llama/Llama-3.1-8B survive.
MCPmcp_servers is not wired in. Hermes brings its own toolsets instead — terminal, file, web, Firecrawl and Playwright by default.
UsageYes, read from the session JSON after the run.
agent_envHERMES_TOOLSETS (default terminal,file,web,firecrawl,playwright), HERMES_MAX_TURNS (default 300), JETTY_HERMES_PROVIDER to force a provider entry.
Hermes has no native provider, so `model_provider` is required. The turn budget is a hard stop: when it fires the agent ends mid-flight, outputs written or not, so raise it for deep-research runbooks rather than hoping. Bedrock goes through Hermes's own plugin with the bearer token, which gets you prompt caching and thinking budgets on that path. YAML ``` agent: hermes model: anthropic/claude-sonnet-5 model_provider: openrouter ``` Pointed at your own server, with a longer budget: JSON ``` { "init_params": { "model": "meta-llama/Llama-3.1-70B-Instruct", "model_provider": "vllm", "agent_env": { "HERMES_MAX_TURNS": "600" } } } ``` * * * Back: [Runtime Configuration](/docs/ai-workloads/runtime-configuration) — the full comparison table, frontmatter defaults, and [per-run overrides](/docs/ai-workloads/runtime-configuration#per-run). --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration/opencode # opencode The open-source SST agent, run as `opencode run`. Headless opencode auto-rejects any permission left on “ask”, and models treat that rejection as a stop signal, so Jetty writes an `opencode.json` that allows everything up front. That file also registers Playwright and Firecrawl MCP servers on every run, Firecrawl with a Jetty-provided key, so browser automation and scraping work without any setup on your side.
Providersanthropic, openai, bedrock, openrouter, cloudflare, vllm
Model idsopencode wants provider/id; Jetty adds the prefix. Keep the vendor namespace on OpenRouter and Cloudflare AI Gateway (anthropic/claude-sonnet-5 on either), bare ids elsewhere, Bedrock's dotted form (anthropic.claude-sonnet-5).
MCPYes. Your mcp_servers are merged on top of the Playwright and Firecrawl presets.
UsageBest-effort. opencode prints text rather than a stable event stream.
agent_env—
`vllm` goes through the AI SDK's OpenAI-compatible loader on the chat completions wire, which makes opencode a good fit for a plain vLLM or SGLang server. YAML ``` agent: opencode model: anthropic/claude-sonnet-5 model_provider: openrouter ``` * * * Back: [Runtime Configuration](/docs/ai-workloads/runtime-configuration) — the full comparison table, frontmatter defaults, and [per-run overrides](/docs/ai-workloads/runtime-configuration#per-run). --- Source: https://jetty.io/docs/ai-workloads/runtime-configuration/pi # pi A small, provider-agnostic agent that ships without an MCP client on purpose. Give it CLI tools and skills. Jetty feeds it the runbook on stdin and pins its config to the run's log volume, so nothing it writes leaks into the workspace it may be committing from.
Providersanthropic, openai, bedrock, openrouter, cloudflare, google, vllm
Model idsPer provider: anthropic/claude-sonnet-5 on OpenRouter and, the same way, on Cloudflare AI Gateway; anthropic.claude-sonnet-5 on Bedrock; gemini-3.1-pro-preview on Google; bare elsewhere.
MCPNo, by design. Declared servers are noted in the agent log and skipped.
UsageYes, including cache tokens.
agent_envOPENAI_WIRE_API for vllm endpoints: chat (default), responses or messages.
`vllm` is registered as a custom provider in pi's `models.json`, so any served id works verbatim. No native provider: name `model_provider`. YAML ``` agent: pi model: anthropic/claude-sonnet-5 model_provider: openrouter ``` * * * Back: [Runtime Configuration](/docs/ai-workloads/runtime-configuration) — the full comparison table, frontmatter defaults, and [per-run overrides](/docs/ai-workloads/runtime-configuration#per-run). --- Source: https://jetty.io/docs/ai-workloads/sandboxes # Sandboxes Every run of a [runbook](/docs/ai-workloads/runbooks) gets its own sandbox: an isolated Linux machine that Jetty creates for that run, runs the agent in, and deletes when the run ends. This page covers what is in a sandbox, what Jetty does to it from start to finish, its limits, and how to run on a Docker image of your own. The short version: - **One sandbox per run.** Nothing carries over between runs, and concurrent runs never share a machine. - **The default is the `python312-uv` snapshot**: Python 3.12, Node 22, Chromium and the common agent CLIs, ready in seconds. - **The agent works in `/app`**, reads uploaded files from `/app/assets` and must write its deliverables to `/app/results`. - **Default size and time:** 4 CPUs, 8 GB of memory, 10 GB of disk, and up to 30 minutes for the agent. - **You can bring your own environment**: a public Docker image, or a base image plus packages. ## What a sandbox is A sandbox is an isolated machine with its own filesystem, processes and network access, created from an image. The agent has a real shell in it: it can install packages, run a browser, start servers and write files, and none of that touches another run or Jetty itself. In most sandboxes the agent runs as root. The sandbox is where the agent runs. The parts of a run that surround the agent run elsewhere: Jetty's worker resolves the run's parameters and secrets, creates the sandbox, streams the agent's log, collects the outputs and records the run. ## Choosing the environment A sandbox is created from one of three sources. Jetty takes the first that applies: | Source | Set with | Start-up | Use it when | | --- | --- | --- | --- | | 1\. Snapshot | `snapshot`, in the runbook's frontmatter or a run's `init_params` | Seconds: the image is already built | Almost always. This is the default | | 2\. Docker image | `image`, with `snapshot` left empty | The image is pulled and prepared before the sandbox starts | You need a toolchain, a runtime or system libraries the default snapshot doesn't have | | 3\. Generated Dockerfile | `base_image`, `apt_packages`, `pip_packages`, with no snapshot and no image | Minutes: the image is built for the run | You need a few extra packages and have no image of your own | A named snapshot wins whenever it exists. If the name doesn't match an available snapshot, Jetty falls through to the image, then to the Dockerfile. `apt_packages` and `pip_packages` apply only on the Dockerfile path: a snapshot or an image already contains its packages. ### The default snapshot A runbook that names no snapshot runs on `python312-uv`. It contains: - Python 3.12 on Debian (bookworm), with `pip`, `uv` and a C build toolchain, so most Python packages install without extra system work - Node 22 and `npm` - `git` and `curl` - Chromium, the Playwright Python library and the Playwright MCP server, for browser automation - The common agent CLIs, pre-installed. See [Runtime Configuration](/docs/ai-workloads/runtime-configuration) for each agent An agent whose CLI isn't in the image is installed when the run starts, which adds a few seconds. The agent can also install what it needs itself, with `pip`, `uv`, `npm` or `apt-get`: a runbook step such as “install `pdfplumber`” works, and costs that install time on every run. ## The life of a sandbox 1. **Prepare.** Jetty resolves the run's parameters, fills the runbook's `{{variables}}`, and downloads the run's attached files. 2. **Create.** A new sandbox is created from the snapshot, image or generated Dockerfile. No secret or user variable is part of the image. 3. **Set up.** Jetty creates the working directories, places the attached files in `/app/assets`, installs or pins the agent CLI, and writes the MCP server configuration if the run has one. 4. **Run the agent.** The agent starts in `/app` with the runbook as its instruction and its [environment](/docs/ai-workloads/secrets-and-environment-variables) applied. Its log is streamed to the run while it works. 5. **Clean up credentials.** When the agent exits, the credential files Jetty wrote are deleted and, if Jetty has checks to run, the processes the agent left behind are stopped. 6. **Collect.** Jetty downloads the agent's log, `/app/results` and `/app/assets`. 7. **Check.** Jetty runs the runbook's [Jetty checks](/docs/ai-workloads/agent-evaluations/code-checks) in the same sandbox, in a clean shell. 8. **Save and delete.** The outputs are saved to the workspace's storage and the sandbox is deleted. A run that fails or times out deletes its sandbox too. As a backstop, a periodic sweep deletes any sandbox that is still running but has gone idle, so a sandbox whose run was interrupted does not linger. ## Where things are | Path | What it holds | Kept after the run | | --- | --- | --- | | `/app` | The agent's working directory | No, apart from the two directories below | | `/app/results` | The run's deliverables, written by the agent | Yes. The frontmatter's `primary_outputs` are surfaced first | | `/app/assets` | Files attached to the run; zips are extracted | Yes | | `/app/checks/` | The git repositories a runbook's code checks declare as sources | No | | `/logs/agent` | The agent CLI's log, session and transcript | Yes, as the run's agent log | A runbook can write these paths as `{{results_dir}}`, `{{assets_dir}}` and `{{checks_dir}}`. Jetty fills them in; a run cannot move them. Important Results are a contract A run whose agent finishes with nothing in `/app/results` fails. Anything the agent leaves elsewhere in the sandbox is gone when the sandbox is deleted. **Getting files in.** Attach files to a run directly, or pre-stage them through the upload API and pass the returned storage paths. The exact calls are in the [machine instructions](/docs/reference/api/machine-instructions). ## Resources and limits | Limit | Default | Change it with | | --- | --- | --- | | CPUs | 4 | `cpus` on the task's step, for an image or Dockerfile sandbox. A snapshot has its own fixed size | | Memory | 8 GB | `memory` on the task's step (for example `"16G"`), under the same rule | | Disk | 10 GB | Fixed | | Agent time | Up to 30 minutes | `timeout_sec` on the task's step, in seconds, to set a shorter budget. For longer jobs, contact [support@jetty.io](mailto:support@jetty.io) | | Agent install | 10 minutes | Fixed. Pre-install the CLI in the image to avoid it | | Sandbox creation | 20 minutes, retried once | Fixed. Covers pulling or building an image | When the agent reaches its time limit it is stopped. A long runbook should write its outputs to `/app/results` as it goes. If a job needs more than 30 minutes, contact [support@jetty.io](mailto:support@jetty.io). ## Network access Sandboxes have outbound network access. The agent can fetch URLs, call APIs, clone repositories and install packages, and it reaches its model provider the same way. Jetty adds no allow-list, so what a run can reach is decided by the credentials you give it; see [Secrets and Environment Variables](/docs/ai-workloads/secrets-and-environment-variables). A run can drive Chromium inside the sandbox for sites that need a real browser. ## Running on your own Docker image When the default snapshot doesn't have what the job needs (a JVM, a GPU-free ML stack, a specific compiler, your company's CLI), build an image and point the run at it. ### What the image needs - **Pullable without credentials.** Jetty takes an image reference and no registry login, so the image must be public (for example on GHCR or Docker Hub). Don't bake secrets into it; they reach the run through the workspace. - **A normal Linux userland**: `bash` and the core utilities. Jetty runs every command through `bash`. - **A way to install the agent.** The agent installers expect a Debian or Ubuntu base (`apt-get`) with `curl`, and install Node if it is missing. Putting Node and the agent CLI in the image skips that work on every run. If the CLI in the image is not the version Jetty pins, Jetty installs the pinned one. - **What your checks use.** If the runbook has Jetty-run code checks, the image needs `python3` for the built-in checks and `git` for check sources. Jetty verifies this before the agent starts and fails the run early if something is missing. The image's `ENTRYPOINT` and `CMD` don't run your job: Jetty starts the sandbox and executes the agent in it. ### Step 1: Build and push the image Dockerfile ``` FROM python:3.12-slim-bookworm # What Jetty and the agent installer rely on: bash, coreutils, curl, git. RUN apt-get update && apt-get install -y --no-install-recommends \ curl git ca-certificates gnupg && \ rm -rf /var/lib/apt/lists/* # Node, so the agent CLI doesn't have to install it on every run. RUN curl -fsSL https://deb.nodesource.com/setup_22.x | sh - && \ apt-get install -y nodejs && rm -rf /var/lib/apt/lists/* # The agent CLI itself (optional; Jetty installs or re-pins it if needed). RUN npm install -g @anthropic-ai/claude-code # Your own toolchain. RUN pip install --no-cache-dir pandas duckdb WORKDIR /app ``` Shell ``` docker build -t ghcr.io/acme/report-env:1.4.2 . docker push ghcr.io/acme/report-env:1.4.2 ``` Use a version tag rather than `latest`, so a run records exactly which environment it used and a new push can't change a runbook's behaviour unnoticed. ### Step 2: Point a run at it An image is a run parameter, not a frontmatter key. Pass `image` and clear `snapshot`, because a named snapshot takes priority and every runbook task has `python312-uv` as its default: Shell ``` curl -X POST "https://flows-api.jetty.io/api/v1/run//" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "init_params": { "snapshot": "", "image": "ghcr.io/acme/report-env:1.4.2" } }' ``` `init_params` merge per top-level key, so the task's runbook, agent, model and variables are unchanged. The run log says which source was used: `Creating sandbox from registry image: …`. ### Step 3: Make it the task's default To use the image on every run, set the same two keys in the task's own `init_params`. Raise `cpus` and `memory` in the step configuration if the job needs more: Task workflow ``` { "init_params": { "instruction": "", "agent": "claude-code", "model": "anthropic/claude-sonnet-5", "model_provider": "openrouter", "snapshot": "", "image": "ghcr.io/acme/report-env:1.4.2", "vars": {}, "file_paths": [] }, "steps": ["run"], "step_configs": { "run": { "activity": "runbook", "agent_path": "init_params.agent", "model_path": "init_params.model", "model_provider_path": "init_params.model_provider", "snapshot_path": "init_params.snapshot", "instruction_path": "init_params.instruction", "template_variables_path": "init_params.vars", "files_path": "init_params.file_paths", "cpus": 4, "memory": "8G" } } } ``` Note A task's parameters are generated from the runbook's frontmatter when the task is built from the runbook text, and the frontmatter has no `image` key. If a task is rebuilt from its runbook, set `image` and the empty `snapshot` again, or pass them with each run. ### Extra packages without an image If all you need is a few packages on top of a stock base, skip the registry and let Jetty generate the Dockerfile. With no snapshot and no image, it builds from `base_image` (default `python:3.13-slim-bookworm`), adds `git`, `curl` and `uv`, then installs your `apt_packages` and `pip_packages`: JSON ``` "init_params": { "snapshot": "", "base_image": "python:3.12-slim-bookworm", "apt_packages": ["ffmpeg", "poppler-utils"], "pip_packages": ["pandas", "pdfplumber"] } ``` This builds an image for the run, which takes minutes. For a runbook you run often, a pushed image starts faster and is the same on every run. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | The run log says `Creating sandbox from snapshot` although you passed `image` | The task's default snapshot still applies | Send `"snapshot": ""` with the image | | Sandbox creation fails or times out | The image is private, the reference is wrong, or the image is very large | Check that `docker pull` works without logging in; slim the image | | `Agent install failed` | The image has no Node and no `apt-get` or `curl` to install it with | Add Node and the agent CLI to the image | | The run fails before the agent starts, naming a missing tool | The runbook's Jetty checks need `python3` or `git` | Add them to the image | | `apt_packages` or `pip_packages` had no effect | The run used a snapshot or an image | Clear `snapshot` and `image`, or install the packages in your image | | The run fails with no results files | The agent wrote outside `/app/results`, or ran out of time | Name `/app/results` in the runbook and write outputs incrementally | --- Source: https://jetty.io/docs/ai-workloads/secrets-and-environment-variables # Secrets and Environment Variables A run needs credentials: a key for its model provider, and tokens for the services the runbook touches. This page describes how Jetty handles those values today: where they are stored, how they get from your workspace into the sandbox, which process receives each one, and what is kept out of the recorded run. The short version: - Values are stored in the **workspace**, or passed with a single run. A runbook holds names, never values. - The sandbox is created **without** your variables. Jetty sets them on the agent's process when it starts the agent. - The agent receives the model provider's key and the secrets the runbook **declares**, not the whole workspace environment. - A secret that Jetty itself uses, for an MCP server or to clone a check repository, is withheld from the agent's shell. - The run's environment is held in memory for the run and is not written to the run record. The agent's own output is recorded as produced. ## Where values come from | Source | Set in | Scope | Use it for | | --- | --- | --- | --- | | Workspace environment | Settings → Integrations and Bring Your Own Key, the `PATCH /api/v1/collections/{c}/environment` endpoint, or the MCP server's `set-environment-vars` tool | Every run in the workspace | Provider keys and integration credentials | | `secret_params` | The run request, or a schedule | One run, or every run of one schedule | A different account or environment for a single run | | Jetty trial keys | Jetty's own secret manager | Workspaces with an active trial | Trial model access before you set a provider key | | `agent_env` | The run request | One run | Non-secret knobs of the agent CLI. It is saved with the run's parameters, so don't put a credential in it | Integration credentials and provider keys share one store: a workspace has a single set of named variables, and the two settings pages are two views onto it. ## How values are stored - **Workspace variables** are stored with the workspace record in Jetty's database as a set of names and values. Jetty does not add its own per-value encryption to them. An update merges: a new name is added, an existing name is replaced, and a name sent with `null` is removed. - **Who can read them.** The environment endpoint returns names and values to any caller with read access to the workspace, and requires write access to change them. The settings page masks each value until you reveal it. Treat membership of a workspace as access to its secrets. - **Schedule secrets.** A schedule's `secret_params` are stored with the schedule. The API accepts them and never returns them. - **Runbooks and tasks** store no values. The `secrets:` block is a list of names, and an MCP server definition that carries a credential-looking header or `env` entry has that entry dropped before it is stored. ## From launch to sandbox 1. **Launch.** When a run starts, Jetty reads the workspace's variables and merges the run's `secret_params` over them. On a name clash the run wins. 2. **Hand-off.** The merged set travels to the worker with the run's launch input. It is not part of the run's parameters and is not saved on the run record. 3. **In the worker.** The worker holds the set in memory as the run's environment. That field is excluded when the run is serialized, so it reaches neither the stored run file nor the database row. The worker logs each name as it is set, with the value masked. One run's environment is never placed in the worker's own process environment, so concurrent runs can't see each other's values. 4. **Sandbox creation.** The sandbox is created from the runbook's image or snapshot with no user variables in it. Nothing is baked into an image, and a snapshot never contains a secret. 5. **Agent start.** Jetty assembles the agent's environment (next section) and applies it to the agent's command when it starts. From there the agent, and every command it runs, reads a variable as it would from a local `.env`. ## What the agent's environment receives The agent does not get the whole workspace environment. Jetty builds its environment in this order, and an earlier entry is not overwritten by a later one, except for the last row: | What | Variables | Condition | | --- | --- | --- | | Provider credentials | The key and settings of the run's model provider, for example `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`, plus what the agent CLI needs to find them | Always, for the provider the run uses. See [Runtime Configuration](/docs/ai-workloads/runtime-configuration) | | Declared secrets | Each `secrets:` entry, under its `env:` name | Set in the run's environment and not withheld ([below](#declared-secrets)) | | Media keys | `REPLICATE_API_TOKEN`, `GEMINI_API_KEY`, `OPENAI_API_KEY` | Forwarded when set, without a declaration, so an agent can call image, video and vision APIs | | `agent_env` | Whatever the run request names | Applied last; overrides the rows above | Any other workspace variable stays out of the sandbox until a runbook declares it. That is why every [integration guide](/docs/integrations) has a “declare the secret” step. ## Declared secrets, and who receives them A `secrets:` entry declares a name and the environment variable the run delivers its value in. Who receives the value follows one rule: **a declared secret reaches the agent unless something of Jetty's consumes it.** RUNBOOK.md frontmatter ``` --- name: nightly-report secrets: # Forwarded to the agent: nothing of Jetty's consumes it. SLACK_BOT_TOKEN: { env: SLACK_BOT_TOKEN } # Consumed by the MCP server below, so withheld from the agent's shell. TRACKER_TOKEN: { env: TRACKER_TOKEN } # Consumed by the code-check source below; also given to Jetty's checks. CHECKS_REPO_TOKEN: { env: CHECKS_REPO_TOKEN, expose_to_checks: true } mcp_servers: tracker: url: https://mcp.tracker.example.com/mcp secret: TRACKER_TOKEN code_checks: sources: - name: checks url: https://github.com/acme/report-checks secret: CHECKS_REPO_TOKEN --- ``` - **An MCP server's secret** becomes that server's `Authorization: Bearer` header in the MCP configuration file the agent CLI reads. It is resolved at run time on a copy of the definition, so the stored run inputs stay token-free. - **A code-check source's secret** authenticates Jetty's clone of that repository. It is passed to git as a host-scoped header in the command's private environment, not on a command line and not in the clone URL. - **Overrides.** `expose_to_agent: true` also gives a consumed secret to the agent, for a runbook body that needs the same token. `expose_to_agent: false` withholds a secret nothing consumes. `expose_to_checks: true` gives a withheld secret to the checks Jetty runs after the agent. - **Who consumes what is decided per run**, from the run's effective `mcp_servers` and `code_checks`, since a run can override either. If a run passes a server's token directly in `mcp_auth_tokens`, that server no longer consumes its secret. - **A declared secret the run did not set** is reported in the run log before the agent starts. The run continues without it. - **Agent-run checks** have the agent's environment and nothing more. A command check the agent runs that reads a withheld secret's variable is refused before the run. See [Code Checks](/docs/ai-workloads/agent-evaluations/code-checks). Important Withholding is hygiene, not a boundary The agent runs as root in its sandbox. Keeping a secret out of its shell stops a command from reading it by accident, and keeps it out of the transcript. It is not a defence against a runbook written to find it: the MCP configuration file is readable by the agent while it runs, and a secret marked `expose_to_checks: true` is used later in a sandbox the agent controlled. Give a run only credentials you would trust the agent with. ## After the agent exits Before anything else reads the sandbox, on every run: 1. **Credential files are deleted**: the MCP configuration and agent auth files that Jetty wrote, so no downloaded artifact carries one. 2. **Leftover processes are stopped** when Jetty has checks to run, so a process the agent left behind can't read the environment of what runs next. 3. **Jetty's checks run in a clean shell**: no login profile, with the `PATH` and `HOME` captured before the agent started. Their environment holds the secrets the agent already had plus those marked `expose_to_checks: true`, and no provider key. 4. **Check output is scrubbed.** The value of every declared secret the run set is removed from check output before it is kept. The sandbox is deleted when the run ends. ## What is and isn't recorded | Where | What Jetty does | | --- | --- | | The run's environment | Never serialized. Not in the run file, not in the database row. | | `secret_params` | Merged into the environment at launch; not stored with the run. | | Run setup | Records the declared secret names, never a value. | | Run log | Names only: which secret was forwarded, which was withheld and why, which was missing. | | Step configuration and model-call parameters | Keys that look like credentials are saved as `[REDACTED]`: `api_key`, `authorization`, `access_token` and similar, and any name ending in `_API_KEY`, `_TOKEN`, `_SECRET_KEY` or `_ACCESS_KEY`. | | Jetty-provided trial keys | Scrubbed from step outputs and error messages, and removed from the run's environment as soon as the step ends. | | Run parameters (`init_params`, `vars`) | **Recorded as sent.** They are inputs, shown on the run page. Never pass a credential as a parameter. | | Agent transcript and output files | **Recorded as produced.** Jetty does not scan the agent's output for your secrets. If the agent prints a token or writes it to a file under its results, it is in the run. | Warning The last two rows are where a credential can end up in a run: a value passed as a parameter, or one the agent prints or writes to its results. Jetty records both as they are, and anyone who can open the run can read them. ## Precedence When the same name is set in more than one place, the first match wins: 1. `agent_env` on the run request 2. `secret_params` on the run request or schedule 3. The workspace environment 4. A Jetty trial key ## Working with it - Store credentials in the workspace and declare them in the runbook. The [Custom Integrations](/docs/integrations/custom) guide walks through it. - Use the `check-secrets` tool to compare what a runbook needs with what a workspace has. It returns names only. - Split workspaces by audience: every member can read a workspace's variables, and every runbook in it can declare them. - Rotate by updating the workspace value. The next run picks it up; a run in progress keeps the value it started with. --- Source: https://jetty.io/docs/ai-workloads/runs # Runs Every time Jetty runs something, such as a [runbook](/docs/ai-workloads/runbooks), it records a **run**: a complete, replayable trace of what happened. This is the observability layer, and it is also the raw material that [evaluations](/docs/ai-workloads/agent-evaluations) read when they evaluate an AI workload. ## Starting a run A deployed runbook can be launched from the web app, from your own agent through the [Jetty plugin](/docs/integrations/bring-your-own-agent), from code with the [SDK](/docs/integrations/sdk) or the [API](/docs/reference/api), or on a cadence with a [schedule](/docs/guides/scheduling-ai-workloads). Below we detail the lifecycle of each runbook run. Every entry point fires the same workload, so runs from all of them land in one history and can be compared like for like. ## The run lifecycle A run is a closed loop that the agent performs with a predefined set of output criteria. At launch, the inputs and the runtime configuration are handed to the agent with the runbook inside a sandbox. The agent works in `/app` and produces a draft. It checks the draft against the runbook's checklist and its own checks, and a fail sends it back to fix and recheck, a loop that is bounded. Then it writes its results and its report, and Jetty runs its own checks against what the agent left. Either way the run is recorded, with its logs, telemetry, validation report, and any labels added afterwards. A run ends with a status (`completed`, `failed` or `cancelled`) and, when it was evaluated, a verdict. The status says whether the job ran to the end; the verdict says whether the work passed. A run whose checks fail still completes, with its outputs kept and the verdict `fail`. See [a failed run and a failed evaluation](/docs/ai-workloads/agent-evaluations#run-vs-verdict). The run includes the following components. ## The parts of a run - **Inputs.** The parameters the run was launched with and any uploaded files, frozen at launch. Uploads are mounted at `/app/assets/`. - **Runtime configuration.** The agent, model and provider, the sandbox snapshot and its resources, and the MCP servers, snapshotted into the record. Secrets are never stored. - **The runbook, as merged.** The exact instruction the agent read, with variables substituted and uploaded files listed. - **Outputs and files.** Everything the agent wrote to `/app/results/`, with the frontmatter's primary outputs surfaced first. A run that ends with an empty results directory fails. - **Logs.** The step-by-step workflow log, streamed live while the run is going and kept afterwards, plus the agent's own transcript. - **Telemetry.** Duration, tokens in and out, API calls, and an estimated cost, recorded per step. - **Evaluations.** The validation report: the agent's entries for its steps, checklist items, agent checks and rubric scores, Jetty's entries for the checks it ran, and the verdict Jetty computes from all of them. See [evaluations](/docs/ai-workloads/agent-evaluations). - **Labels.** Key/value tags added after the fact, by you, by Pelly, or through the API. For each step, the run also records the activity that ran and how long it took, the inputs it received (resolved from [path expressions](/docs/reference/api/steps#path-expressions)), the outputs it produced, and its status (running, completed, failed, cancelled). You can view a run in the app's run viewer or use Jetty's agent tools to access it for further investigation. --- Source: https://jetty.io/docs/ai-workloads/agent-evaluations # Agent Evaluations An evaluation is a test of an agent's work. Once the agent has done its job, an evaluation looks at what it produced and answers one question: does this work meet the standard set for it? In Jetty, evaluations are written into the [runbook](/docs/ai-workloads/runbooks), next to the instructions, and Jetty carries them out as part of every [run](/docs/ai-workloads/runs). Their results add up to the run's verdict: pass or fail. ## Why agents need evaluations Start with an ordinary program. Give it the same input twice and it does exactly the same thing twice. If it worked yesterday, it works today, so a test you ran once tells you something that stays true. An agent does not behave that way. It is driven by a language model, and the model picks each next step by sampling from many plausible ones. Give an agent the same instructions and the same input twice and it can take a different path each time: open files in a different order, reach for a different tool, phrase its answer differently. Most of those paths end somewhere good, and some do not. That is what **non-deterministic** means: the result of a run is one draw from a range of possible results, not a fixed answer. This changes what “it works” means. One good run shows that the agent _can_ do the job, not that it _will_ do it next time. Nor can you take the agent's word for it: an agent that says it succeeded is reporting what it believes, and a model can be confident and wrong at the same time. To know whether a run did the job, something has to examine that run's actual output. ## Why evaluate every run If every run can turn out differently, then every run has to be checked. So Jetty does not evaluate an agent once and trust it afterwards: it evaluates each run while the run is happening, against that run's own outputs. This gives you three things. - **A verdict on every result.** You learn whether a particular output is good before you rely on it, not after something downstream breaks. - **Coverage of inputs nobody planned for.** A fixed test set only contains the cases someone thought of in advance. An evaluation that runs with the job judges the input the job actually received, including ones no one anticipated. - **A record to learn from.** Every run leaves a verdict and the evidence behind it. Across many runs you can see how often the agent succeeds, which checks it tends to fail, and whether a change to the runbook made things better or worse. That is how an agent's work improves through experience: each run is both a piece of work and a measurement of how well it was done. ## Rubric and programmatic evaluation A runbook is written in one of two patterns, named by the `evaluation` key in its frontmatter. The pattern decides how the agent grades its own output in the evaluate step and what it iterates on. Both patterns carry the same [types of evaluation](#types) (code checks, a checklist, and in a rubric, judge scores) and write the same [report](#report). The create-runbook skill picks one: rubric by default, programmatic when the task clearly has right and wrong answers. | | `evaluation: rubric` | `evaluation: programmatic` | | --- | --- | --- | | For | Output whose quality has several dimensions: reports, content, summaries, images, training material | Output with a right answer: structured data, extraction, code, API results | | The evaluate step | Score the output on each criterion of a table, 1–5, with what 5, 3 and 1 look like | Give each output `PASS`, `PARTIAL` or `FAIL` against criteria written in the runbook | | The bar | An overall average of at least 4.0 and no criterion below 3 | Every output passes | | In the report | One `judge` entry per criterion, with its score and a threshold of 3; the average is checked by a checklist item | Step, checklist and code check entries; no judge entries | | Iteration | Rework the lowest-scoring criteria, up to three rounds | Rerun the failed items with targeted fixes, up to three rounds | A rubric is the agent scoring itself, so treat its scores as the agent's opinion and back the dimensions that matter with code checks or a separate grader. See [Rubrics](/docs/ai-workloads/agent-evaluations/judges#rubric). ## The types of evaluation A runbook can carry four types of evaluation. They differ in who carries them out and in how far the result can be trusted, not in how much they count: each one lands in the run's validation report as an entry, and every entry counts toward the verdict. | Type | What it is | Who runs it, and when | In the fix-and-recheck loop? | Can the agent skip or misreport it? | | --- | --- | --- | --- | --- | | [Agent check](/docs/ai-workloads/agent-evaluations/code-checks#agent) | A check that needs the agent's tools or live session: the issue exists in Linear, the server it started answers. | The agent, before it writes its report | Yes: a failure sends the agent back to fix the output | It can leave one out, which is recorded as `skipped` | | [Jetty check](/docs/ai-workloads/agent-evaluations/code-checks#jetty) | A command or a built-in check run against the outputs: the files exist, the JSON parses, the totals reconcile. | Jetty, in the sandbox, after the agent has finished | No: it runs after the agent has finished | No | | [Checklist](/docs/ai-workloads/agent-evaluations/checklists) | Plain-language conditions the agent confirms by inspection: the summary names every flagged receipt. | The agent, before it writes its report | Yes: a failed item sends the agent back to fix the output | It can leave one out, which is recorded as `skipped` | | [Judge](/docs/ai-workloads/agent-evaluations/judges) | A model or agent handed the output and a criterion, returning a score or verdict with evidence: tone, completeness, relevance. | The agent scoring itself against a rubric, a judge model the runbook calls, or a separate grader run | Yes for a self-scored rubric: a low score sends the agent back to rework it. A separate grader is outside the loop | Only a separate grader is out of the agent's reach | Agent checks and Jetty checks are the two kinds of [code check](/docs/ai-workloads/agent-evaluations/code-checks), written under `## Code Checks` and told apart by who runs them. Choosing between them is a choice between a result the agent can act on and a result that can be trusted. ### Prefer an agent check to let the agent fix the output Prefer an agent check when a failure should be fixed rather than only recorded. The agent runs it inside its fix-and-recheck loop, so when it fails the agent reworks the output and runs it again before it writes its report. An agent check is either an instruction the agent carries out or a shell command marked [`executor=agent`](/docs/ai-workloads/agent-evaluations/code-checks#executor), which the agent runs itself. The price is trust: the result is the agent's word, and a check it leaves out is recorded as `skipped`. ### Prefer a Jetty check to trust the result Prefer a Jetty check wherever a command can see what has to be checked: it is deterministic, costs no tokens, and does not depend on the agent under test. It runs outside the fix-and-recheck loop, after the agent has finished, so the agent cannot skip it or claim a result it did not earn. The same fact means a failure cannot be fixed in that run: the output stays as the agent left it, and the failure goes into the verdict. A condition that matters can have both: an agent check, so the agent catches and fixes the problem in the loop, and a Jetty check with the same command, so the verdict does not rest on the agent's report. ## How the report comes together Every evaluation in a run ends up as an entry in one file, `validation_report.json` in the results directory. The agent writes it first, and Jetty completes it after the agent has finished. 1. **The agent works and runs its checks.** Jetty appends a note to the runbook naming the checks the agent runs and the ones Jetty runs. The agent runs its agent checks and checklist items, fixing and rechecking within its iteration limit. 2. **The agent writes its report and finishes.** The report holds one entry per step, checklist item, agent check and rubric criterion it scored. It has no entries for Jetty checks, and any `verdict` the agent writes is only a hint. Jetty stops every process the agent left behind, then collects the results directory with the report as the agent left it. 3. **Jetty runs its checks.** It restores the uploaded assets from its own copy, clones the git sources fresh, and runs each [Jetty check](/docs/ai-workloads/agent-evaluations/code-checks#jetty) in turn against the outputs. Every result is also saved, with up to 16 KB of output, as `code_checks.json`. 4. **Jetty merges its results into the report.** - Jetty adds its own entries after the agent's, one per Jetty check, each marked as run by Jetty. - It drops any entry the agent wrote for a Jetty check or for an id the runbook never declared, and any entry it cannot read, such as one missing its `kind`. The run log names each one dropped. - If the agent wrote no report, Jetty writes a fresh one. If the file is not a JSON object, Jetty sets it aside as `validation_report.agent-invalid.json` and writes a fresh one. - If the sandbox could not be prepared for the checks, every Jetty check becomes an `error` entry naming why, so a run whose checks could not run is never reported as passing. The agent's outputs are still saved. 5. **Jetty fills the gaps.** It adds a `skipped` entry for every agent check and checklist item the report leaves out, so the saved report lists every declared check. 6. **Jetty computes the [verdict](#verdict)** from the merged report and records it on the run, alongside a summary of the counts and failed checks. It logs a mismatch if the agent's own verdict disagreed. In a runbook with no Jetty checks, step 3 runs nothing and step 4 adds no entries of Jetty's. The agent's report is still cleaned and filled in the same way, and still decides the verdict. ## Entry statuses Each entry in the report has one of four statuses: - `pass`: the check held. - `fail`: the check ran and the outputs did not satisfy it. A judge entry also fails when its score is below its threshold. - `error`: the check could not be carried out: it timed out, its command could not run, it was malformed, or the sandbox could not be prepared for it. - `skipped`: an agent check or checklist item that the agent was to report and left out. A Jetty check is never skipped. The [Code Checks](/docs/ai-workloads/agent-evaluations/code-checks#results) page lists exactly when a code check gets each one. ## The verdict From the finished report, Jetty computes one verdict for the run. The agent's own verdict is never used. | Verdict | When | | --- | --- | | `fail` | Any check failed: a `fail` or `error` on a code check (agent or Jetty) or a checklist item, or a judge score under its threshold. Under [strict evaluation](#strict), a `skipped` entry too. | | `pass` | At least one check passed, and none failed. | | `unknown` | Nothing was decided: the runbook declares no checks, the agent wrote no report and Jetty ran no checks, or the report is the old, pre-v2 shape. | - **No partial credit, and no ranking between types.** One failed checklist item fails the verdict as surely as a failed Jetty check. What differs between types is how far you can trust a `pass`, not how much a `fail` counts. - **Step entries never count.** The `step` entries that record the agent's progress through the runbook are shown but do not affect the verdict. ## Strict evaluation An agent can finish without reporting every check it was given: it ran out of turns, it forgot one, or it left out a check it was failing. Jetty records each one as `skipped`, with the message “Declared in the runbook but not reported by the agent”, so a declared check never disappears. What a skipped check does to the verdict is the runbook's choice: - **By default** (`strict_evaluation: false`), a skipped check is shown but does not count. The verdict is decided by the checks that did report. - **Under strict evaluation** (`strict_evaluation: true`), a skipped check counts as failed, so the verdict fails. A report that is missing or cannot be read leaves every declared check the agent was to report as skipped, and so fails it too. Only agent checks and checklist items can be skipped. A Jetty check always produces a result, so strict evaluation changes nothing for a runbook with only Jetty checks and no checklist. Turn it on once a runbook's agent checks are settled, so that an agent cannot pass by leaving out the check it would fail. YAML ``` --- strict_evaluation: true # a skipped check fails the verdict --- ``` Like the code checks' [frontmatter settings](/docs/ai-workloads/agent-evaluations/code-checks#sources), it is copied into the task when the runbook is saved, and a single run can override it by passing `strict_evaluation` in its `init_params`. The run's evaluation records whether it was strict. ## A failed run and a failed evaluation Every run answers two separate questions. Its **status** says whether the job ran to the end. Its **verdict** says whether the work is good. A **failed run** is one where the job could not be carried out, so there is nothing to evaluate. A **failed evaluation** is a run that did the job and produced outputs that fell short of the bar. Jetty keeps them apart: a failing check never fails the run, and a failed run has no verdict. | Run | Verdict | What it means | | --- | --- | --- | | `completed` | `pass` | The job ran and every evaluation held. | | `completed` | `fail` | The job ran and its outputs are saved, but at least one evaluation failed. This is the run to investigate: the eval view lists the failed checks first. | | `completed` | `unknown`, or none | The job ran but nothing was evaluated: a runbook with no checks, an old-style report, or a task saved before [evals v2](#evals-version), which has no verdict at all. | | `failed` | none | The job could not be carried out. The run's error says why, and the eval view says the run failed before it was evaluated. | | `cancelled` | none | The run was stopped on request before it finished. | A run fails, rather than completing with a failed verdict, when: - a required input has no value, or the runbook has no instruction; - a git source for the code checks is unreachable with its credential or its ref does not exist, caught before the sandbox starts; - the sandbox cannot be created, or the agent cannot be installed or started; - the sandbox session runs past Jetty's hard session limit; - the agent writes nothing to `/app/results`. This is usually missing or invalid model-provider credentials: the agent could not run at all; - the platform itself hits an error running the step. An agent that is stopped at the run's time limit _after_ writing some outputs has not failed the run: the run completes, and the checks decide whether what it left is good enough. ## Where to read them - **In the app**, the run list and the run viewer show the run's status, and the eval view shows the verdict with each check, failed ones first. - **Through the API**, a run's record carries `status` and `error`, and its verdict as `attributes.verdict` (`pass`, `fail` or `unknown`), with `attributes.evaluation` beside it: the counts per type, the failed checks, whether the run was strict, and the rubric's `overall_score`. The full list of entries is in the runbook step's `outputs.evaluation.checks`. - **In the run's files**, `validation_report.json` holds every entry, and `code_checks.json` the full record of each Jetty check. - **In webhooks**, a run event carries the run's status, not its verdict. Fetch the run to read the verdict. One detail for API readers: a run that fails inside the runbook step can be stored with `status: completed`, with the step's `outputs.success` set to `false` and its `outputs.error` saying why. The app shows these runs as failed. To find every failed run, check both the status and the step's `success`. ## Which runs are evaluated Everything on this page applies to **evals v2** tasks: a task saved from the runbook editor once evals v2 is on, from the current runbook templates. A task saved before that runs as it always did: Jetty runs none of its code checks and computes no verdict, and the agent's own report stands as written. An evals v2 task whose runbook declares no `## Code Checks` and no `## Checklist` gives Jetty nothing to run, and an agent that writes an old-style report (no `"version": 2` and no `checks` list) leaves the verdict resting on its own claim. The run still goes ahead in both cases, and the run log says which one happened. --- Source: https://jetty.io/docs/ai-workloads/agent-evaluations/code-checks # Code Checks A code check tests a run's outputs and answers pass or fail. It is written in the [runbook](/docs/ai-workloads/runbooks), under `## Code Checks`. There are two kinds, told apart by who runs them, and a run takes them in this order: - [Agent checks](#agent) are run and reported by the agent while it works, for what only the agent can reach. - [Jetty checks](#jetty) are run by Jetty after the agent has finished, so the agent cannot skip them, change them or misreport them. Both count the same in the verdict. Prefer a Jetty check wherever a command can see what has to be checked: its result does not depend on the agent under test. ## Writing a check Each check is a level-three heading followed by one fenced block. The heading carries an id and a name separated by a dash. The fence's language says what the check is, and an `executor=` word after it says who runs it. One check of each kind, in the order a run takes them: Markdown ```` ## Code Checks ### issue-filed — The Linear issue exists and is Done ```agent executor=agent Use the linear MCP server to fetch the issue whose key is in {{results_dir}}/issue.txt. Pass if it exists and its state is Done; fail otherwise, and say which state it is in. ``` ### api-up — The API the agent started answers ```bash executor=agent curl -fsS http://localhost:8000/health ``` ### outputs-exist — Every required output is present ```bash executor=jetty test -s {{results_dir}}/summary.md && test -s {{results_dir}}/output.json ``` ### output-is-json — output.json parses ```yaml executor=jetty use: json_valid path: output.json ``` ```` `executor=agent` is the only choice for an `agent` fence and `executor=jetty` the default for a command, so both can be left off. The fence lines Jetty accepts: | Fence line | The check | Who runs it | | --- | --- | --- | | ` ```agent ` or ` ```agent executor=agent ` | [Instruction check](#agent-fence) | The agent, always | | ` ```bash executor=agent ` | [Command the agent runs](#executor) | The agent | | ` ```yaml executor=agent ` | Built-in check | The agent. Accepted, but the agent has no copy of the built-in's code, so leave built-ins to Jetty. | | ` ```bash `, ` ```sh `, ` ```shell ` or a bare ` ``` `, optionally with `executor=jetty` | [Inline command](#inline) | Jetty | | ` ```yaml ` or ` ```check `, optionally with `executor=jetty` | [Built-in check](#builtin) | Jetty | - The **id** is letters, digits, `.`, `_` and `-`. It names the check in the report and the eval view, so keep it stable. The **name** is free text: say what passing means. - Only the first fence under a heading counts. `executor=` is case-insensitive, and any other word on the fence line is ignored. - A check that cannot run is recorded as `error` with the reason, never silently dropped: a heading with no usable fence, an unknown language or built-in, an empty command, a `{{checks_dir}}` path for an undeclared source, a template blank such as `{TODO: …}`, ` ```agent executor=jetty `, a misspelt executor (`executor=jety`, `executor:agent`), or ` ``` executor=agent ` with no language. ## Agent checks ### Instruction checks Some facts cannot be read from a run's files: the issue exists in Linear, the page renders in a browser, the row landed in the database. Checking them takes a tool, and an MCP server in particular needs an MCP client, which only the agent has. Write the check as an instruction in an `agent` fence: say what to do, with which tools, and what passing means. The MCP servers a check needs are declared in the frontmatter, under `mcp_servers` or `code_checks.mcp_servers`. A definition never carries a token: it names a `secrets:` entry with `secret:`, and Jetty puts the value into the server's configuration when the run starts (see [Secrets](#secrets)). A credential-looking key written into a definition is dropped with a warning. Authenticated MCP servers reach the agent on the claude-code and antigravity runtimes. ### Commands the agent runs A command that needs the agent's live session, such as a server the agent started, is marked `executor=agent` and run by the agent before it reports. It names paths with the same `{{results_dir}}`, `{{checks_dir}}` and `{{assets_dir}}` placeholders as any other check; Jetty fills them in before the agent reads the runbook. It runs with the agent's environment, which does not have the variables Jetty gives its own checks. Markdown ```` ### summary-written — summary.md is not empty ```bash executor=agent test -s {{results_dir}}/summary.md ``` ```` - A command that reads a secret the agent does not receive is refused before the run, as an `error`. - When one uses `{{checks_dir}}`, Jetty clones the [git sources](#sources) before the agent starts. ### How agent checks are reported Jetty tells the agent the id of every check it must run, and the agent records one `code_check` entry per id in its report, with `pass` or `fail` and a one-line message. Jetty does not re-run it. An entry for an id the runbook never declared is dropped, and a declared check the report leaves out is recorded as `skipped`. See [how the report comes together](/docs/ai-workloads/agent-evaluations#report). ## Jetty checks ### How Jetty checks run Jetty checks are not part of the agent's fix-and-recheck loop. The agent is told that Jetty runs them after it finishes and not to run them itself, so it never sees their result. A failed Jetty check shows in the run's verdict, not in another round of work. Once the agent has finished, Jetty stops every process it left behind, so nothing can change the outputs, and removes its credential files. It restores the uploaded assets from its own copy, clones the git sources fresh, and runs each check in turn in the sandbox the agent worked in, against the outputs exactly as the agent left them: the same files Jetty saves. Each check gets a profile-less shell with the `PATH` the sandbox had before the agent ran. It sees the outputs, the assets, the sources and the [secrets given to the checks](#secrets), and not the agent's model credentials, MCP servers or session. ### Inline commands The fence is the command. It runs from `/app` in `bash` with `-e` and `pipefail` set, so every line of a multi-line fence must succeed. Exit `0` is pass; any other exit is fail, except the ones that mean the check could not be carried out (see [Results](#results)). The last non-empty line of output becomes the check's message. Paths are written with the `{{results_dir}}`, `{{checks_dir}}` and `{{assets_dir}}` placeholders, and the run's own `{{name}}` inputs are filled in as they are in the rest of the runbook. A run cannot change these paths. For a script that reads them from its environment, a Jetty check also has them as variables: | Variable | Value | What is there | | --- | --- | --- | | `RESULTS_DIR` | `/app/results` | The run's outputs, exactly as the agent left them. | | `CHECKS_DIR` | `/app/checks` | One directory per [git source](#sources), cloned fresh. | | `ASSETS_DIR` | `/app/assets` | Files uploaded with the task, restored from Jetty's copy. | | `CHECK_ID` | the heading's id | Handy when one script serves several checks. | ### Built-in checks The common cases need no code. A `yaml` fence names a built-in with `use:` and its parameters, all required. Paths and globs are relative to the results directory; a `pattern` is a Python regular expression, matched line by line. Missing or unusable parameters are an `error`, not a failure. | use | Parameters | Passes when | | --- | --- | --- | | `file_exists` | `path` | The file exists. | | `min_size` | `path`, `bytes` | The file exists and is at least that many bytes. `bytes` is a whole number (`10KB` is refused). | | `json_valid` | `path` | The file parses as JSON. | | `regex_present` | `glob`, `pattern` | At least one matching file contains the pattern. | | `regex_absent` | `glob`, `pattern` | No matching file contains the pattern. A glob that matches no file fails, so a typo in the glob cannot pass. | | `markdown_relative_links_resolve` | `path` | Every relative Markdown link in the file points at an existing file. Web, `mailto:` and `#anchor` links are not checked. | ### How Jetty checks are reported Jetty adds one `code_check` entry per check after the agent's entries, marked as run by Jetty, with the command, the exit code, the last 2 KB of output and how long it took. `code_checks.json` keeps up to 16 KB of output for each. An entry the agent wrote for a Jetty check is dropped, and if the sandbox could not be prepared, every Jetty check is an `error` naming why. See [how the report comes together](/docs/ai-workloads/agent-evaluations#report). ## Bringing code with the check When a check outgrows one line, keep it as a script and point the check at it: - **Upload it with the task.** Files attached to the task are at `/app/assets`: `python3 {{assets_dir}}/validate.py {{results_dir}}/output.json`. - **Declare a git source** in the frontmatter. Jetty clones it to `/app/checks/` at the named branch, tag or commit (the default branch otherwise). The URL is a plain `https://` address. A private repository names the secret to clone with; the token goes to git alone and never appears in the clone, the agent's environment or any log. YAML ``` --- secrets: GITHUB_TOKEN: {} # clones the source below LINEAR_TOKEN: {} # authenticates the linear server mcp_servers: linear: url: https://mcp.linear.app/sse secret: LINEAR_TOKEN # names the secret, never the token code_checks: timeout_sec: 120 # per check; default 120, max 900 sources: - name: checks # → /app/checks/checks type: git url: https://github.com/acme/output-checks ref: v1.4.0 # a branch, a tag or a commit secret: GITHUB_TOKEN # only for a private repository --- ``` These keys are defaults. When the runbook is saved as a task, Jetty copies `mcp_servers` (with `code_checks.mcp_servers` folded in), `code_checks` and `strict_evaluation` into the task's `init_params`. A run may override any of them, one top-level key at a time: a run that passes `code_checks` replaces the whole block. `secrets` is never copied: it declares which values the run must bring. Jetty confirms every source is reachable with its credential, at its ref, before the sandbox starts, so a wrong URL, a missing ref or an expired token fails the run in seconds. For Jetty checks the clone happens after the agent has exited, so the agent cannot have changed the check code. ## Secrets A runbook names the secrets a run needs under `secrets:`; it never holds their values. Each value comes from the workspace's environment variables, or from the run's `secret_params`, and is never stored with the task. An entry can say which environment variable the value arrives in, and who receives it: YAML ``` secrets: OPENAI_KEY: {} # env var OPENAI_KEY; the agent receives it PUBLISH_TOKEN: env: GH_PUBLISH_TOKEN # the env var it arrives in (default: its name) expose_to_agent: false # keep it out of the agent's environment expose_to_checks: true # but give it to the checks Jetty runs ``` | A secret that is | The agent | Jetty checks | | --- | --- | --- | | Named by nothing of Jetty's | receives it | receive it | | Named with `secret:` by a git source or an MCP server | does not: git or the server's configuration gets it instead | do not, unless the entry sets `expose_to_checks: true` | | Marked `expose_to_agent: true` | receives it | receive it | | Marked `expose_to_agent: false` | does not | do not, unless the entry sets `expose_to_checks: true` | - A declared secret the run does not supply is named in the run log, and the run goes on without it. An MCP server whose secret is missing is configured without a token; a source whose secret is missing usually fails the reachability check, with a hint naming it. - A run may pass a server's token directly, under `mcp_auth_tokens` (`{"linear": "…"}`); the secret is then not used for that server. - The value of every declared secret the run set is scrubbed from each check's output before it is kept. - Keeping a secret out of the agent's environment is hygiene, not a security boundary: Jetty checks run in the sandbox the agent had, so a secret given to them is trusted to that sandbox. ## Results Each code check ends with one of four statuses: | Status | When | | --- | --- | | `pass` | The command exited 0, the built-in held, or the agent reported its check as passing. | | `fail` | The check ran and the outputs did not satisfy it. | | `error` | The check could not be carried out: it ran past its timeout, its command was not found or not executable (exit 126 or 127), a built-in's parameters were unusable, the sandbox could not be prepared (a missing `git` or `python3`, a failed clone), or it was malformed in one of the ways [listed above](#writing). | | `skipped` | An agent check the report leaves out. A Jetty check is never skipped. | `fail` and `error` fail the run's verdict, and so does `skipped` under [strict evaluation](/docs/ai-workloads/agent-evaluations#strict). See [the verdict](/docs/ai-workloads/agent-evaluations#verdict). ## Tips - **Test a new check against a run that passed.** A wrong check fails the verdict of every run from then on, and you stop trusting the verdict. - **Print the reason.** `test -s file || echo "summary.md is empty"` reads better in the eval view than a bare exit code. - **Keep checks fast.** They run one after another, each under its own timeout. A check that needs the network or a slow tool is usually better as a script in a source repository, where it can be tested on its own. - **Do not depend on what the agent installed.** The sandbox image and the runbook's `pip_packages` / `apt_packages` are what a check can rely on; `python3` and `git` are on the default image. --- Source: https://jetty.io/docs/ai-workloads/agent-evaluations/checklists # Checklists A checklist is a plain list of conditions the agent walks through before it reports done, written in the [runbook](/docs/ai-workloads/runbooks) next to the steps. The agent ticks through the list, fixes what fails, rechecks, and only then writes its report. ## Writing a checklist Put the items under a `## Checklist` heading, one checkbox per line. A qualifier on the heading, such as `## Checklist (manual)`, is allowed, and so is the templates' numbering (`## Step 8: Checklist`). - Only checkbox lines count (`- [ ]` or `- [x]`). Prose between them, and anything inside a fenced block, is not an item. - Each item's id is its text in lower case with every run of other characters turned into a dash: `No placeholder text remains` becomes `no-placeholder-text-remains`. Two items with the same id are told apart as `-2`, `-3`. - A run's `{{name}}` inputs are filled into the items before their ids are taken, so the ids match what the agent read. ## How a checklist runs The checklist is the agent's own evaluate step. When an item fails, the agent goes back, fixes the output, and rechecks the list. The loop is bounded: the templates allow three rounds, after which the agent keeps its best result, notes what still fails, and writes its report. ## How items are reported Jetty tells the agent the id of every item, and the agent records one `checklist` entry per id in its report, with `pass` or `fail` and a one-line message. Jetty does not re-check an item. A failed item fails the run's verdict; an item the report leaves out is recorded as `skipped`. See [how the report comes together](/docs/ai-workloads/agent-evaluations#report). ## Tips - **Make each item small and observable.** “The CSV has one row per input image” can be confirmed; “the output is good” cannot. - **Keep item text stable.** Editing an item changes its id, so runs before and after the edit record it as different items. - **Move what a command can see to a Jetty check.** A file exists, a pattern is absent: as a [Jetty check](/docs/ai-workloads/agent-evaluations/code-checks#jetty), the agent cannot misreport it. --- Source: https://jetty.io/docs/ai-workloads/agent-evaluations/judges # Judges A judge is handed an output to assess, an instrument to apply (a yes/no question, a criterion, or a 1–5 scale) and optionally the source context. It returns a structured verdict with cited evidence, so the answer is both a value and an explanation: “out of range: a 40% tip on a $12 bill.” A rubric is a set of these instruments with thresholds. ## Writing a rubric A runbook written with `evaluation: rubric` carries its rubric as a table of criteria, each scored 1–5, with what 5, 3 and 1 look like, and a pass threshold (see [rubric and programmatic evaluation](/docs/ai-workloads/agent-evaluations#rubric-programmatic)). Write each column so two readers would give the same score: name what is present or missing, not how good it feels. ## How judges run - **Self-scored rubric.** The agent scores its own output against the rubric in its evaluate step. Below the bar, it reworks the lowest-scoring criteria and scores again, up to three rounds, then keeps its best version and notes what is still weak. - **LLM judge.** A single model call, fast and cheap enough to run in flight: the runbook calls a judge model through the credentials minted for the run, hands it only the output and the criterion, and gets a verdict back. Because the context is narrow, a capable agent can be paired with a much smaller judge model. The `simple_judge` [action](/docs/reference/api/steps) is this shape as a workflow step, scoring 1–5 or against a categorical rubric. - **Agent judge.** A separate agent run, usually a grader runbook, with tools: it opens the output files, checks them against sources, and works through the rubric the way a careful reviewer would. It is slower and costs more, and the agent under test never sees it. The [Jetty and Friends](/docs/jetty-and-friends) examples use this shape to grade agents running outside Jetty. ## How scores are reported Each rubric criterion becomes a `judge` entry in the agent's report. An entry whose score is below its threshold fails the run's verdict. The overall average is checked by a [checklist](/docs/ai-workloads/agent-evaluations/checklists) item (“Overall rubric average is at least 4.0”), and the run's evaluation records it as `overall_score` against a `pass_threshold` of 4.0. See [how the report comes together](/docs/ai-workloads/agent-evaluations#report). ## Tips - **Treat a self-scored rubric as the agent's opinion.** Agents love giving themselves a pat on the back. Back the criteria that matter with a [Jetty check](/docs/ai-workloads/agent-evaluations/code-checks#jetty) where a command can see them, and with a separate agent judge where it takes a reader. - **Give the judge only what it needs.** The output and the criterion, not the agent's whole transcript: a narrow context keeps the verdict about the output. --- Source: https://jetty.io/docs/reference/api # Jetty REST API Reference The Jetty REST API is one HTTP surface. The primary way in is the **tasks API**: deploy a runbook as a task, start runs of it, and poll the results. An OpenAI-compatible chat-completions endpoint is also provided _for compatibility_, so OpenAI-shaped SDKs and tools can point at Jetty without changes. A handful of smaller APIs cover scheduling, async callbacks, and source control. Everything here authenticates the same way and records every call as a [run](/docs/ai-workloads/runs) (the API calls them trajectories). ## Base URL & auth The base URL is `https://flows-api.jetty.io`. Authenticate every request with a bearer token: Shell ``` Authorization: Bearer $JETTY_API_TOKEN ``` Grab a token from [Settings → API keys](/settings/api-keys), or run the `jetty-setup` skill from an agent with the Jetty plugin to write it to `~/.config/jetty/token`. Prefer not to hand-roll requests? The [typed SDK](/docs/integrations/sdk) and the [MCP tool layer](/docs/integrations/mcp-server) wrap everything below. | Endpoint | Method | What it does | | --- | --- | --- | | `/api/v1/run/{workspace}/{task}` | POST | **The primary endpoint.** Start a run of a deployed task; async — returns an id you poll. | | `/api/v1/tasks/{workspace}/{task}` | GET / PUT | Read and update a deployed task's configuration. | | `/v1/chat/completions` | POST | OpenAI-compatible _compatibility layer_: passthrough LLM proxy or a runbook run, depending on the body. | | `/api/v1/routines/{workspace}/{task}` | GET / POST / PATCH / DELETE | CRUD and lifecycle for scheduled runs. | | `/api/v1/trajectories/{workspace}/{task}` | GET | List and get runs to monitor progress. | | `webhook_url` (run field) | POST to you | Completion callback for async runs. There is no webhooks endpoint; you pass a URL when you start a run or create a schedule. | | `/v1/github/pull-requests` | POST | Open a GitHub PR from a workflow. | ## Tasks API The tasks API is the front door. Deploy a runbook as a task once (from the web app or your agent), and it becomes a stable, named thing you can run, schedule, and override per run — without resending the runbook each time: Shell ``` curl -X POST https://flows-api.jetty.io/api/v1/run/my-workspace/cbc-homepage-summary \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "init_params": { "vars": { "url": "https://www.cbc.ca/" } } }' ``` The call is async by design: it returns an id, and you poll the run for progress and results. The `init_params` merge over the task's stored defaults ([caller wins](/docs/ai-workloads/runtime-configuration)). The merge is shallow: each top-level key you send replaces the stored one whole. Sending `vars` replaces the task's entire `vars` object, so include every variable the runbook needs, not just the ones you're changing. The body also takes `secret_params` (injected as environment variables, never stored in the run) and `webhook_url` / `webhook_secret` (see [Webhooks](#webhooks)). For short jobs, `/api/v1/run-sync/{workspace}/{task}` waits for the result instead. The full task surface is in the [machine instructions](/docs/reference/api/machine-instructions). ## Chat Completions API The `POST /v1/chat/completions` endpoing enables the OpenAI API so you can point at Jetty without changes. It is secondary to the tasks API. There are two modes. ### Passthrough mode (no `jetty` block) Without a `jetty` block, the endpoint is a standard LLM proxy across 100+ providers. Send it like any OpenAI chat request, and every call is recorded as a run you can replay and grade later. Shell ``` curl https://flows-api.jetty.io/v1/chat/completions \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.6-terra", "messages": [ { "role": "user", "content": "Summarize the theory of relativity in 3 bullets" } ] }' ``` ### Runbook mode (add a `jetty` block) Add a `jetty` block with `runbook: true`, a `collection` and a `task` name, and the same endpoint provisions a sandbox, runs an agent against your runbook, and returns structured results. The runbook markdown travels inline as the system message, and the user message is passed to the runbook as `{{prompt}}`. There is no URL field: to run a runbook from a URL, fetch it and send it inline. The task is created in the workspace on the first call. The `collection` field takes your workspace name. Shell ``` RUNBOOK=$(curl -s https://raw.githubusercontent.com/org/repo/main/RUNBOOK.md) jq -n --arg runbook "$RUNBOOK" '{ model: "anthropic/claude-sonnet-5", messages: [ { role: "system", content: $runbook }, { role: "user", content: "Research the EV charging market" } ], jetty: { runbook: true, collection: "my-workspace", task: "ev-market-research", template_variables: { region: "north-america" } } }' | curl https://flows-api.jetty.io/v1/chat/completions \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d @- ``` `template_variables` fill the runbook's other `{{vars}}`. Runs that go past the sync window (1200 s by default, set with `jetty.timeout_hint`) return `202` with `jetty_metadata.poll_url`. Add `jetty.webhook_url` to get a `202` straight away and a [webhook](#webhooks) when the run finishes. See the full walkthrough in the [get-started guide](/docs/get-started). ## Scheduling API Scheduled runs are configured under the routines endpoint (the API's name for a schedule). Create, list, update, and delete them, or trigger one immediately with run-now. Shell ``` POST /api/v1/routines/{workspace}/{task} # create GET /api/v1/routines/{workspace}[/{task}] # list PATCH /api/v1/routines/{workspace}/{task}/{name} # update DELETE /api/v1/routines/{workspace}/{task}/{name} # delete POST /api/v1/routines/{workspace}/{task}/{name}/run-now # run now ``` A schedule stores `init_params_overrides` (same shallow merge as a run; keys must be declared in the task's `init_params`, or the call is a 400) and can carry its own `webhook_url` / `webhook_secret`, which apply to every run it fires. ## Webhooks Pass `webhook_url` when you start a run and Jetty POSTs to it once the run finishes, whether it completed or failed. It's the alternative to polling. There is no registration endpoint. The URL rides on the request: - `POST /api/v1/run/{workspace}/{task}`: top-level `webhook_url` and `webhook_secret` (JSON or multipart fields). - `POST /v1/chat/completions` in runbook mode: `jetty.webhook_url` and `jetty.webhook_secret`. With a webhook set (and `stream` off) the call returns `202` straight away instead of waiting for the run. - Schedules: `webhook_url` and `webhook_secret` on the routine, applied to every run it fires. `/api/v1/run-sync` doesn't send webhooks. It ignores both fields, because the response already carries the result. Shell ``` curl -X POST https://flows-api.jetty.io/api/v1/run/my-workspace/cbc-homepage-summary \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "init_params": { "vars": { "url": "https://www.cbc.ca/" } }, "webhook_url": "https://ops.example.com/jetty", "webhook_secret": "'"$WEBHOOK_SECRET"'" }' ``` ### Signed vs. unsigned `webhook_secret` is optional. Setting a URL is enough to get a delivery. The secret only decides whether the delivery is signed: | Header | With a secret | Without a secret | | --- | --- | --- | | `Content-Type: application/json` | Sent | Sent | | `X-Mise-Timestamp` | Sent (Unix seconds) | Sent (Unix seconds) | | `X-Mise-Trajectory-Id` | Sent | Sent | | `X-Mise-Signature` | Sent: hex `HMAC-SHA256(secret, "{timestamp}.{body}")` | **Absent** | Unsigned deliveries are fine for a throwaway endpoint or a test. For anything that acts on the payload, set a secret and make your receiver **fail closed**: reject a request with no `X-Mise-Signature` instead of treating the header as optional. Otherwise anyone who learns the URL can forge a delivery. To verify, recompute the HMAC over the timestamp header, a literal `.`, and the raw request body (not re-serialized JSON), compare it in constant time, and reject stale timestamps to stop replays: Python ``` import hashlib, hmac, time def verify(headers, raw_body: bytes, secret: str, max_age=300) -> bool: sig = headers.get("X-Mise-Signature") ts = headers.get("X-Mise-Timestamp") if not sig or not ts: return False # fail closed on unsigned deliveries if abs(time.time() - int(ts)) > max_age: return False expected = hmac.new(secret.encode(), f"{ts}.".encode() + raw_body, hashlib.sha256).hexdigest() return hmac.compare_digest(expected, sig) ``` ### Payload & delivery - **Body**: the full run (trajectory) JSON, the same document `GET /api/v1/trajectory/{workspace}/{task}/{trajectory_id}` returns. Check its `status` (`completed` or `failed`). Your `webhook_url` appears in it, but `webhook_secret` never does. - **Retries**: up to 3 attempts with exponential backoff (1 s, then 2 s) and a 30 s timeout each. Any response below `400` counts as delivered. - **Best effort**: a failed delivery is logged on the run but never fails the run. Poll the run if you need a guarantee. This callback is separate from the `webhook` [step activity](/docs/reference/api/steps), which is an outbound HTTP call a workflow makes mid-run. ## GitHub PR API Open pull requests programmatically from a workflow, so an agent that produces code or config changes can land them through review rather than commit directly. ## Monitoring a run List or get runs to track one from start to finish. Each run captures the step inputs and outputs, so you can watch progress, inspect results, and grade them. Shell ``` curl https://flows-api.jetty.io/api/v1/trajectories/my-workspace/cbc-homepage-summary \ -H "Authorization: Bearer $JETTY_API_TOKEN" ``` ## Behind the endpoint One API surface, five components: - **Passthrough proxy**: the OpenAI-compatible front door to 100+ providers. - **Workflow engine**: runs multi-step DAGs. See [Running AI Workloads](/docs/product-overview/running-ai-workloads). - **Runbook engine**: spins up sandboxed agents to execute markdown runbooks. - **Persistence**: a relational database for metadata plus object storage for artifacts. - **Tracing**: every execution is recorded as a [run](/docs/ai-workloads/runs) — the full trace of what ran, in what order, with which inputs and outputs. That's what makes a run replayable and gradeable. ### Durable execution Workflows are backed by a durable execution engine, which gives you automatic retries and exactly-once semantics. Each runbook run gets its own isolated sandbox, so concurrent runs can't see or step on each other. --- Source: https://jetty.io/docs/reference/api/machine-instructions # Machine instructions The machine-level counterpart to [Running AI Workloads](/docs/product-overview/running-ai-workloads): the endpoints, payloads, and config schemas an agent or programmatic caller needs to create tasks, run runbooks, and read the results. The [API reference](/docs/reference/api) covers the chat-completions endpoint in depth. Base URL `https://flows-api.jetty.io`, header `Authorization: Bearer $JETTY_API_TOKEN`. Tokens resolve everywhere in the same order: explicit value → `JETTY_API_TOKEN` env → `~/.config/jetty/token`. The full OpenAPI spec is unauthenticated at `flows-api.jetty.io/openapi.json`. ## Creating tasks A task is a deployed runbook: named, addressable as `{workspace}/{task}`, runnable. The API and SDK name a workspace `collection`. Text ``` POST /api/v1/tasks/{workspace} # 201; body: {name, description, workflow, is_private, has_file_uploads} PUT /api/v1/tasks/{workspace}/{task} # partial update; bumps revision GET /api/v1/tasks/{workspace}/{task} DELETE /api/v1/tasks/{workspace}/{task} ``` `GET` works on any public task, and the instruction inside the returned workflow is the runbook markdown — every public runbook is also a template. The `workflow` for a runbook task is one step running the `runbook` activity, with every execution field indirected through `init_params`: JSON ``` { "init_params": { "agent": "claude-code", "model": "anthropic/claude-sonnet-5", "model_provider": "openrouter", "snapshot": "python312-uv", "instruction": "", "vars": {}, "file_paths": [] }, "step_configs": { "run": { "activity": "runbook", "agent_path": "init_params.agent", "model_path": "init_params.model", "snapshot_path": "init_params.snapshot", "instruction_path": "init_params.instruction", "template_variables_path": "init_params.vars", "files_path": "init_params.file_paths", "cpus": 4, "memory": "8G", "timeout_sec": 1800 } }, "steps": ["run"] } ``` The `*_path` indirection is the point: because the step reads its config through paths into `init_params`, any of these can be overridden per run without redeploying. SDK: `createTask(collection, name, workflow, description?)` / `updateTask`. ## Running runbooks Text ``` POST /api/v1/run/{workspace}/{task} # async; 200 {workflow_id, metadata} POST /api/v1/run-sync/{workspace}/{task} # blocks; returns the trajectory POST /v1/chat/completions # OpenAI-compatible; `jetty` block selects runbook mode ``` Run endpoints accept JSON or multipart. JSON body fields: `init_params`, `secret_params`, `webhook_url`, `webhook_secret`; multipart adds repeatable `files`. One of these, `secret_params`, is parsed from the raw body and doesn't appear in the published OpenAPI schema, so don't trust `openapi.json` as exhaustive here. (Trial keys are applied automatically when the workspace is eligible — there is no flag.) The chat-completions form flips into runbook mode with `jetty.runbook: true`, and the runbook markdown travels inline as the system message — there is no URL-reference field; to run a runbook from a URL, fetch it and send it inline. The full `jetty` block: `runbook`, `collection`, `task`, `agent`, `model_provider`, `snapshot`, `timeout_sec`, `cpus`, `memory`, `template_variables`, `agent_env`, `mcp_servers`, `file_paths`, `files`, `webhook_url`, `webhook_secret`, `timeout_hint`. When a run outlasts the sync window (default 1200 s, tunable via `jetty.timeout_hint`) it returns 202 with `jetty_metadata.poll_url` and `logs_url`. With `jetty.webhook_url` set (and `stream` off) it returns that 202 straight away. `run-sync` ignores `webhook_url` and `webhook_secret`: it sends no callback. An async run responds immediately with `workflow_id: "{workspace}-{task}--{trajectory_id}"`. The run id (the API's `trajectory_id`) is the short hex suffix after the last `--` — poll `GET /api/v1/trajectory/{workspace}/{task}/{trajectory_id}` or stream live logs from `GET /api/v1/workflows-logs-stream/{workflow_id}` (SSE). A synchronous run that exceeds ~100 s returns a Cloudflare 524 while the run continues server-side: treat it as “still working”, poll the run, never retry the POST — a retry starts a second run. Chat-completions without a `jetty` block is passthrough mode: a plain LLM proxy routed across 100+ providers, every call recorded as a run. OpenRouter routing is tunable per workspace via the `OPENROUTER_PROVIDER_ORDER` and `OPENROUTER_ALLOW_FALLBACKS` environment variables. SDK equivalents: `runWorkflow`, `runWorkflowSync`, `runWithFiles`, `runAndWait(collection, task, initParams, {pollMs: 2000, timeoutMs: 1_800_000})`. ## Reading runs Text ``` GET /api/v1/db/trajectory/{c}/{t}/{id} # one trajectory GET /api/v1/db/trajectories/{c}/{t} # list (page, limit, status, include_archived) POST /api/v1/db/trajectories/{c}/{t}/query # filtered list: label_key/label_value, param_key/param_value, status, time range GET /api/v1/db/stats/{workspace}/{task} # aggregate numbers GET /api/v1/file/{storage_path} # one artifact (paths come from step outputs) GET /api/v1/trajectory/{c}/{t}/{id}/download # everything as a zip ``` The list response is wrapped as `{trajectories, total, page, limit, has_more}`, and a run's steps are an object keyed by step name, not an array. Label filtering goes through the `query` endpoint, not the plain list. Labels are `key`/`value`/`author` triples attached per run — the raw material for cohorts, sweeps, and eval results (`graded: pass`, `config: warm`, `cohort: 2026-08`). ## Defining inputs and initial parameters `init_params` merge shallowly, caller over task definition: deploy-time values are defaults, run-time values override. Shallow means per top-level key: a run that sends `vars` replaces the task's whole `vars` object, so send every variable the runbook uses, not just the changed ones. Schedule `init_params_overrides` merge the same way. Every `*_path` value is a [path expression](/docs/reference/api/steps#path-expressions). Inside a step, each parameter resolves in priority order: literal in step config → `*_path` in step config → literal in `init_params` → `*_path` in `init_params` → declared default. `GET /api/v1/tasks/{c}/{t}/init-params-schema` returns the declared keys with defaults and type hints — use it to build forms or validate calls. Template variables ride in `init_params.vars`; attached files always land in `init_params.file_paths` (the list exists on every run, empty or not). ## Files in and out Attach files as multipart `files` on the run endpoints (`runWithFiles` in the SDK), or pre-stage with `POST /api/v1/sandbox/upload` (up to 10 files, 50 MB each) and pass the returned storage paths. Uploads land at `init_params.file_paths[]` — a list, not named keys — and are mounted at `/app/assets/` in the sandbox; zips are auto-extracted. Everything the agent writes to `/app/results/` is persisted to workspace storage as `results_files`, with the frontmatter's `primary_outputs` surfaced first. Fetch via `GET /api/v1/file/{storage_path}` or the run zip download above. ## Environment variables and secrets Workspace-level env vars are set via Settings or `PATCH /api/v1/collections/{c}/environment` (merge semantics; a `null` value deletes the key). They form the run's environment, but the sandbox does not receive all of them: the agent process gets the model provider's credentials plus the env vars the runbook declares in its `secrets:` frontmatter block (`NAME: { env: VAR }`). An undeclared workspace var never reaches the agent. Per-run `secret_params` merge over the workspace's env at launch (the run wins) and are never persisted to the run. A declared secret that an MCP server or a `code_checks.sources[]` entry names with `secret:` is consumed by Jetty and withheld from the agent's shell; `expose_to_agent: true|false` on the entry overrides either default, and `expose_to_checks: true` gives a withheld secret to the checks Jetty runs after the agent. A declared secret the run does not set is logged and the run proceeds without it. The `secrets:` block also powers the plugin's `check-secrets` preflight (`{configured, missing, ready}`, names only). Full path: [Secrets and Environment Variables](/docs/ai-workloads/secrets-and-environment-variables). ## Attaching MCP servers Runbook tasks declare MCP servers in the step config, as a local command or a remote HTTP endpoint: JSON ``` "mcp_servers": { "playwright": { "command": "npx", "args": ["@playwright/mcp@latest"] }, "figma": { "url": "https://mcp.figma.com/mcp", "type": "http" } } ``` Bearer tokens for HTTP servers come from `mcp_auth_tokens` (per-server) or `mcp_auth_token` (all servers), both of which accept [path expressions](/docs/reference/api/steps#path-expressions) so a token can flow in from `init_params` or an earlier step. MCP support varies by runtime: claude-code, antigravity, and opencode wire the config in; codex, hermes, and pi currently ignore it. ## Sweeping agents and models There is no first-class sweep primitive — but the `*_path` indirection makes one unnecessary for small matrices: the same task runs under any `{agent, model, model_provider}` combination passed through `init_params`. TypeScript ``` for (const cfg of [ { agent: "claude-code", model: "anthropic/claude-sonnet-5", model_provider: "openrouter" }, { agent: "codex", model: "gpt-5.5", model_provider: "openai" }, { agent: "opencode", model: "anthropic/claude-sonnet-5", model_provider: "cloudflare" }, ]) { const run = await jetty.runAndWait("acme", "triage", { ...cfg, vars: { ticket } }); await jetty.addLabel("acme", "triage", run.trajectory_id, "sweep.agent", cfg.agent, "sweep"); } ``` Label each run with its configuration, then compare via the labeled run list or `get-stats`. For server-side fan-out, the `list_emit_await` activity maps a list into parallel child runs (`max_concurrency`, `continue_on_error`) and `extract_from_trajectories` aggregates the results. Keep the grader fixed while sweeping — same evaluation task across every combination, or the numbers don't compare. ## Defining runbooks Frontmatter is the machine-readable contract: | Field | Values | Notes | | --- | --- | --- | | `version` | semver string | bumped by the `optimize-runbook` skill on applied edits | | `evaluation` | `rubric` | `programmatic` | how the run self-checks | | `agent` | `claude-code` (default), `codex`, `antigravity`, `opencode`, `hermes`, `pi` | the runtime | | `model`, `model_provider` | provider-qualified slug + gateway (`anthropic`, `openai`, `bedrock`, `openrouter`, `cloudflare`, `google`, `vllm`) | must agree with each other | | `snapshot` | e.g. `python312-uv` | pre-built sandbox image | | `timeout_sec` | 60–1800 (30 min) | agent wall-clock budget | | `primary_outputs` | file list, relative to results dir | surfaced first in the run | | `secrets` | `{NAME: {env, description, required}}` | env vars the run expects | The body must contain the objective, the output manifest, a `## Parameters` table for `{{vars}}`, numbered steps, an evaluation step, a bounded iteration step, and a final verification checklist — the structure the `create-runbook` skill scaffolds and validates. Deploying is `createTask` with the workflow shape above; the transformation is invertible, so a task's runbook can always be recovered from `init_params.instruction`. ## Timeouts - `timeout_sec` (step config or frontmatter): the agent's execution budget. Runs are supported up to 1800 s (30 min). On expiry the agent process is killed and the step fails. For longer jobs, contact [support@jetty.io](mailto:support@jetty.io). - Sync HTTP calls: ~100 s at the edge before a 524. The run continues; poll, don't retry. - Chat-completions sync window: `jetty.timeout_hint`, default 1200 s, then 202 + poll URL. - Client-side: SDK `runAndWait` gives up polling after 30 min by default (`timeoutMs`), matching the 30-minute run budget. Long-running runbooks should write output incrementally so partial results survive a timeout. ## Snapshots A snapshot is a pre-built sandbox image. Cold-building an environment (base image + apt + pip + agent install) costs minutes per run; a snapshot cuts sandbox bring-up to seconds and pins the toolchain. Environment resolution order is snapshot → custom `image` → generated Dockerfile (`base_image` + `apt_packages` + `pip_packages` — these three apply only on the Dockerfile path; a snapshot already contains its packages). The default snapshot is `python312-uv` (Python 3.12 + uv, with Chromium for browser automation). The snapshot rides through `init_params`, so a single task can be pointed at a different environment per run. To run on a custom `image`, send it with an empty `snapshot`: a task built from a runbook defaults to `python312-uv`, and a named snapshot wins. Lifecycle, layout, limits and the custom-image walkthrough are in [Sandboxes](/docs/ai-workloads/sandboxes). ## Webhooks Pass `webhook_url` on an async run (`/api/v1/run`), a chat-completions runbook run (`jetty.webhook_url`) or a schedule, and Jetty POSTs to it once the run completes or fails. It's the alternative to polling. There is no registration endpoint, and `run-sync` never sends one. - **Payload**: the full run (trajectory) JSON. Check its `status`: `completed` or `failed`. It never includes `webhook_secret`. - **Headers, always**: `Content-Type: application/json`, `X-Mise-Timestamp` (Unix seconds), `X-Mise-Trajectory-Id`. - **Signed vs. unsigned**: `webhook_secret` is optional. With it, the POST also carries `X-Mise-Signature` = `HMAC-SHA256(secret, "{timestamp}.{raw body}")`, hex-encoded. Without it the delivery still goes out, with no `X-Mise-Signature` header at all. - **Verifying**: receivers that act on the payload should fail closed. Reject a request with no signature, compare it in constant time, and reject stale timestamps. - **Delivery**: up to 3 attempts, exponential backoff, 30 s per attempt; any status below 400 counts as delivered. Webhook failure never fails the run. Verification code and the full header table are in the [API reference](/docs/reference/api#webhooks). Distinct from the `webhook` _step activity_, which is an outbound HTTP call made mid-workflow. ## Scheduling runs A schedule fires a task on a cadence. Same workflow as a manual run — same steps, same config, same evals; there is no separate scheduled code path — each firing producing its own run tagged with `triggered_by_routine_id`. The API keeps the older name, _routine_, in its paths and fields. Text ``` POST /api/v1/routines/{workspace}/{task} # create GET /api/v1/routines/{workspace}[/{task}] # list (includes live next_run_at) PATCH /api/v1/routines/{c}/{t}/{name} # update any subset POST /api/v1/routines/{c}/{t}/{name}/pause|resume|run-now GET /api/v1/routines/{c}/{t}/{name}/runs # trajectories this schedule fired ``` Create body: JSON ``` { "name": "nightly-eval", "cadence": { "type": "daily", "hour_utc": 9, "minute_utc": 0 }, "init_params_overrides": { "vars": { "mode": "full" } }, "webhook_url": "https://ops.example.com/jetty", "overlap_policy": "skip" } ``` Cadence is a structured enum — `manual | hourly | daily | weekdays | weekly` with `hour_utc`, `minute_utc`, and `day_of_week` — not raw cron. `manual` saves the configuration as a run-now preset with no schedule. `init_params_overrides` are validated against the task's declared `init_params` (unknown keys are a 400). `overlap_policy` defaults to `skip`: a firing is dropped rather than stacked if the previous run is still going. Pausing preserves the schedule; `run-now` bypasses it and returns a normal `workflow_id`. The two standing use cases: recurring jobs (daily report, weekly digest, scheduled data pull) and regression detection — a runbook that passed at 84% last month can quietly regress under model drift, and a [schedule](/docs/guides/scheduling-ai-workloads) plus a webhook turns that into an alert instead of a discovery. * * * Next: the conceptual treatment in [Running AI Workloads](/docs/product-overview/running-ai-workloads), the chat-completions endpoint in the [API reference](/docs/reference/api), or the catalog of pre-built [actions](/docs/reference/api/steps). --- Source: https://jetty.io/docs/reference/api/steps # Actions Jetty supports a number of pre-built actions that work alongside [runbooks](/docs/ai-workloads/runbooks). A runbook hands a whole job to an agent. An action does one well-defined piece of work with no agent in the loop: call a model, fan out over a list, fetch a file, grade an output. Put the two together and a single run can prepare inputs before a runbook starts, hand the runbook its data, and score what it produced afterwards. An action is one entry in a workflow's `step_configs`. Each entry names an `activity` — the unit of work Jetty knows how to run — plus the config that activity needs, and the workflow's `steps` list sets the order. The `runbook` activity is itself an action, which is how a [runbook task](/docs/reference/api/machine-instructions) becomes a one-step workflow and how other actions slot in before and after it. Actions pass data to one another through [path expressions](#path-expressions). Actions fall into four families: AI models, control flow, data processing, and evaluation. The tables below list a representative slice of each, not the whole registry. ## Path expressions Actions don't share variables. They read each other's data through **path expressions**: dot-notation strings that point at a value somewhere in the run. Any config field ending in `_path` takes a path instead of a literal value, so `prompt_path: "init_params.user_prompt"` means “use the value found at that path as this step's prompt.” | Pattern | Points at | | --- | --- | | `init_params.` | An input to the run, as passed by the caller or defaulted by the task. | | `.outputs.` | Something an earlier action produced. `steps..outputs.` is the explicit form of the same path. | | `.inputs.` | The resolved input an earlier action received. | | `…[0].` | One element of an array output. | | `…[*].` | Every element of an array output, collected into a list. | Wire a later action to an earlier one by pointing its `*_path` at `earlier_step.outputs.…`; the order in `steps` guarantees the source has produced its output first. JSON ``` { "init_params": { "user_prompt": "Explain quantum computing simply" }, "step_configs": { "analyze": { "activity": "litellm_chat", "model": "gpt-4o", "prompt_path": "init_params.user_prompt" }, "grade": { "activity": "simple_judge", "content_path": "analyze.outputs.text", "criteria": "Clear, correct, and under 200 words" } }, "steps": ["analyze", "grade"] } ``` `analyze` reads its prompt from the run's inputs and `grade` reads the text `analyze` produced. Pass a different `user_prompt` at run time and both steps follow, with nothing in the workflow to edit. ### Where a value comes from A parameter can be set in more than one place. Inside a step it resolves in priority order: literal in step config → `*_path` in step config → literal in `init_params` → `*_path` in `init_params` → the activity's declared default. `init_params` themselves merge shallowly, caller over task definition. That is why a deployed runbook task reads its agent, model, and instruction through `*_path` fields, and why the same task can be rerun with a different agent, model, or input without being redeployed. The full rules are in the [machine instructions](/docs/reference/api/machine-instructions). ### Magic variables inside a fan-out When `list_emit_await` fans out over a list, each child run can reach a few magic variables through template expressions, written as `{{ expression }}`: | Variable | Meaning | | --- | --- | | `$item` | The current item. | | `$index` | The current index, 0-based. | | `$parent_id` | The parent run's trajectory id. | | `$item_len` | The length of the current item. | | `$item_type` | The type name of the current item. | Path expressions show up in a few other places too. A bearer token for an attached MCP server (`mcp_auth_token`) accepts one, so the token can come from `init_params` or an earlier step. The inputs recorded on every [run](/docs/ai-workloads/runs) are the values these paths resolved to. And because the wiring between actions is just paths, Jetty can draw any workflow as a graph: each action a node, each path reference an edge. The live syntax reference is `GET /api/v1/step-templates/path-expressions`. ## AI models These call language, image, and video models. Each family needs the matching provider key configured; see [runtimes](/docs/ai-workloads/runtime-configuration) and [Integrations](/docs/integrations) for how Jetty resolves credentials at run time. | Category | Activity | What it does | | --- | --- | --- | | AI models | `litellm_chat` | One chat completion against any of 100+ providers via LiteLLM. | | AI models | `litellm_batch` | Many completions in one step: fan out a list of requests. | | AI models | Gemini steps | Prompt and reader activities plus `gemini_image_generator`. | | AI models | AWS Bedrock | Invoke models hosted on Bedrock. | | AI models | `replicate_text2image` | Replicate models for text-to-image and the rest of the Replicate catalog. | ## Control flow These shape how a workflow runs: branching, looping, fanning out across many inputs, and pulling data back out of prior runs. | Category | Activity | What it does | | --- | --- | --- | | Control flow | `list_emit_await` | Fan out over a list and run children in parallel. | | Control flow | `conditional_branch` | Take one path or another based on a value. | | Control flow | `number_sequence_generator` | Produce a numeric range to iterate over. | | Control flow | `loop_counter` | Track iterations in a loop. | | Control flow | `extract_from_trajectories` | Pull values out of earlier runs into the current run. | ## Data processing The glue steps: moving text and files around, attaching metadata, and reaching out over HTTP. | Category | Activity | What it does | | --- | --- | --- | | Data processing | `text_concatenate` | Join text fragments into one string. | | Data processing | read / download / save file steps | Read, fetch, and persist files through Jetty storage. | | Data processing | metadata | Attach or read metadata on an asset. | | Data processing | `webhook` | Call out to an external HTTP endpoint mid-workflow. | ## Evaluation These grade and analyze runs, the steps that drive [run-based evaluation](/docs/ai-workloads/agent-evaluations). | Category | Activity | What it does | | --- | --- | --- | | Evaluation | `simple_judge` | LLM-as-judge: score on a 1–5 scale or a categorical rubric. | | Evaluation | `select_trajectories` | Filter prior runs into a set to analyze. | | Evaluation | `visualize_correlation` | Plot how scores or signals relate across runs. | ## Discovering actions programmatically This catalog is a snapshot, while the live registry is the source of truth. The API calls actions step templates: list the full set from your editor with `list-step-templates` (MCP) or `listStepTemplates` ([SDK](/docs/integrations/sdk)), then inspect any one to see its exact config shape. * * * Ready to wire these together? See [writing runbooks](/docs/ai-workloads/runbooks#writing-runbooks) for the runbook-first path, or [machine instructions](/docs/reference/api/machine-instructions) for the programmatic workflow surface. --- Source: https://jetty.io/docs/integrations # Integrations A workload rarely runs alone. It gets triggered by someone or something, reads from document repositories, CRMs and databases, and reports back out into the world — a Slack channel, an email, your own application. In Jetty those connections are **integrations**, and under the hood every one is an environment variable: a named value injected into the sandbox at run time, which the agent reads exactly the way it would read a local `.env`. Here are a few rules of thumb for what belongs where: - Talking to a model? AI Gateway key - Connecting to anything else? Integration key - Changes each run? Input parameter ## Connect a service One guide per service, each built around a worked example you can run: how to mint the credential with the smallest scope, where to store it, and the runbook that uses it. [GitHubGive a runbook a fine-grained token for one repository, store it in the workspace, then push a branch and open a pull request from every run.](/docs/integrations/github)[Google DriveGive a runbook its own OAuth credentials for Google Drive, choose the right scope, then write files into a folder and read the files other people put there.](/docs/integrations/google-drive)[SlackGive a runbook its own Slack bot token with just chat:write, invite it to a channel, then post messages and confirm each one by its permalink.](/docs/integrations/slack)[Custom IntegrationsConnect a service that has no guide of its own: store its credential in the workspace, declare it in the runbook, and call the API or attach its MCP server.](/docs/integrations/custom) ## How an integration reaches a run 1. **Set it once.** Add the variable in **Settings → Integrations** ([/settings/integrations](/settings/integrations)) — a name and a value, like `SLACK_WEBHOOK_URL` or `DATABASE_URL`. 2. **The runbook declares it.** A runbook lists the variables it needs in the `secrets:` block of its frontmatter. Every run of that runbook in the workspace then gets them in its sandbox environment; nothing to pass per run. 3. **The runbook names it.** The steps say what to do and which variable to use — “post the summary to the channel behind `SLACK_WEBHOOK_URL`” — and the agent picks it up from the environment like any CLI tool would. No guide for your service? [Custom Integrations](/docs/integrations/custom) is the general recipe. For the full path a value takes, see [Secrets and Environment Variables](/docs/ai-workloads/secrets-and-environment-variables). > **Secrets are never written to logs.** Injected variables are available to the run but kept out of run output and logs. An agent can confirm what's present with the `check-secrets` tool without revealing any values. ## What lives in an integration Anything that isn't a model-provider key: the credentials for the tools your agent calls, service URLs, standing configuration. | Integration | Typical variables | The workload uses it to | | --- | --- | --- | | [Slack](/docs/integrations/slack) | `SLACK_WEBHOOK_URL` or `SLACK_BOT_TOKEN` | Post summaries and results; alert on a failing run. | | [Google Drive](/docs/integrations/google-drive) | `GDRIVE_CLIENT_ID`, `GDRIVE_CLIENT_SECRET`, `GDRIVE_REFRESH_TOKEN` | Write reports into a shared folder; read the files people drop there. | | Postgres | `DATABASE_URL` | Query and reconcile records as part of the job. | | [GitHub](/docs/integrations/github) | `GITHUB_TOKEN` | Read private repos, open pull requests. | | HubSpot / Salesforce | `HUBSPOT_TOKEN`, `SALESFORCE_TOKEN` | Read and update CRM records. | | Email service | `RESEND_API_KEY` | Send the report when the run completes. | | Your own application | `APP_BASE_URL`, `APP_API_TOKEN` | Call internal APIs and webhooks. | One current limit: there is no OAuth flow — an integration is a long-lived credential, so use tokens, webhooks and connection strings. Rotating one is a single edit: update the value, and the next run picks it up. ## Integrations among the other knobs Four kinds of configuration reach a run, and they differ by where they're set and how long they stand: | Configuration | Where it's set | Scope | Read by | | --- | --- | --- | --- | | **Integration** | Settings → Integrations | Every run in the workspace | The agent and the tools it calls | | **AI Gateway key** | Settings → Bring Your Own Key | Every run in the workspace | The [runtime](/docs/ai-workloads/runtime-configuration), to reach its model provider | | **Parameter** (`{{vars}}`) | The run request | One run | Substituted into the runbook body before the agent starts | | **Runtime knob** (`agent_env`) | The run request | One run | The agent CLI itself ([per-run overrides](/docs/ai-workloads/runtime-configuration)) | --- Source: https://jetty.io/docs/integrations/github # Connecting a Jetty Runbook to GitHub This guide shows how to let a Jetty runbook push commits to a GitHub repository and open pull requests. The worked example is a runbook that writes a short haiku, commits it as `haikus/.md` (for example `haikus/1792614032.md`) on a new branch, and opens a pull request for it. When told to, it pushes the commit straight to a branch instead. You can reuse the same setup for any runbook that changes code, regenerates files, updates documentation or otherwise needs to land its work in a repository. Credential `GITHUB_TOKEN` A fine-grained token for one repository Permissions Contents, Pull requests Read and write on both, nothing else Token lifetime 366 days at most Then every run fails with 401 until it is renewed Each run A branch and a pull request Or a push straight to the base branch The steps Set up GitHub 1. [Step 1: Pick the destination repository](#step-1) 2. [Step 2: Create a fine-grained personal access token](#step-2) Configure Jetty 1. [Step 3: Store the token in your Jetty workspace](#step-3) 2. [Step 4: Declare the secret in your runbook](#step-4) Create and test the runbook 1. [Step 5: Hand the token to git without writing it down](#step-5) 2. [Step 6: Create the task](#step-6) 3. [Step 7: Run a test](#step-7) **Time needed:** about 15 minutes, one time. **Prerequisites:** - A GitHub account with write access to the destination repository - A Jetty workspace where you can set environment variables and create tasks - `curl` on your machine (only for the one-time token check) ## How it works Runbooks run in an isolated sandbox that has `git` but no GitHub login. You give the runbook one credential: | Variable | Description | | --- | --- | | `GITHUB_TOKEN` | A fine-grained personal access token limited to the destination repository, allowed to read and write its contents and its pull requests | During each run, the runbook tells git to read the token from the environment, clones the repository over HTTPS, pushes a branch, and opens the pull request through the GitHub REST API. `GITHUB_TOKEN` is the name Jetty's [integrations](/docs/integrations) page uses for GitHub, and the name the `gh` CLI picks up by itself if your sandbox image has it. Where the token travels: minted once on GitHub, stored once in the workspace, admitted by the runbook's `secrets:` block, and read from the sandbox environment on every run by git and by the REST call. Its value never lands in the runbook, a URL, a config file or a log. **Permissions.** The token gets two repository permissions. GitHub adds a third on its own, and a fourth is needed only for Actions files: | Repository permission | Access | Needed for | | --- | --- | --- | | Contents | Read and write | Clone and push | | Pull requests | Read and write | Open the PR | | Metadata | Read-only | Added by GitHub on its own | | Workflows | Read and write | Only if your runbook edits files under `.github/workflows/` | The token can't reach any other repository, and it can't change settings, secrets or Actions workflows. Note Why not a deploy key, or your own `gh` login? An SSH deploy key can push, but it can't open a pull request. The token your `gh` CLI holds is a classic OAuth token with the `repo` scope over every repository you can reach, far more than one runbook needs. A fine-grained token scoped to one repository is the smallest credential that does the whole job. Teams that want a credential not tied to a person can use a GitHub App instead: its installation tokens last an hour and are minted per run, at the cost of a key-exchange step this guide doesn't cover. ## Set up GitHub Steps 1 and 2 are done once, on GitHub. ### Step 1: Pick the destination repository Create a repository or pick an existing one, and note its `owner/name` and its base branch (`main`, usually). Two recommendations: 1. **Protect the base branch** (**Settings > Rules** or **Branches** > require a pull request before merging). The runbook can then propose changes but never land them unreviewed. The example runbook opens a pull request by default for exactly this reason. 2. **The base branch needs at least one commit.** An empty repository has no branch to clone, so add a README first. ### Step 2: Create a fine-grained personal access token 1. On GitHub, open your profile > **Settings > Developer settings > Personal access tokens > Fine-grained tokens > Generate new token** (direct link: [github.com/settings/personal-access-tokens/new](https://github.com/settings/personal-access-tokens/new)). 2. **Token name:** something like `jetty-github-runbook`. **Expiration:** pick a date and write it down (see the warning below). 3. **Resource owner:** the account or organization that owns the repository. 4. **Repository access:** **Only select repositories**, then pick the destination repository. 5. **Permissions > Repository permissions:** set **Contents** to **Read and write** and **Pull requests** to **Read and write**. Leave everything else at _No access_. (Metadata: Read-only is selected for you.) 6. Click **Generate token** and copy it. It starts with `github_pat_` and is shown once. Warning This step decides when scheduled runs stop working. Fine-grained tokens always expire. GitHub's default policy caps them at 366 days, and an organization can set a shorter cap. When the token expires, every run fails at its first GitHub call with `401 Bad credentials`. Put the expiry date in your calendar: renewing is this step again, plus updating `GITHUB_TOKEN` in [Step 3](#step-3), one tool call away if your coding agent has the Jetty MCP server. Note Organization repositories When the resource owner is an organization, the organization's settings decide whether fine-grained tokens are allowed at all and whether an owner has to approve each one. A token waiting for approval behaves like one with no access: the repository comes back `404`. Owners approve tokens under the organization's **Settings > Personal access tokens**. ## Configure Jetty ### Step 3: Store the token in your Jetty workspace Add `GITHUB_TOKEN` as an environment variable on your Jetty workspace. Any of these three ways works. **Web app.** Open **Settings > Integrations** and add `GITHUB_TOKEN`. **Coding agent.** If you work through the Jetty plugin for Claude Code or the [Jetty MCP server](/docs/integrations/mcp-server) in Cursor, VS Code, Windsurf or Zed, save the token in a file of its own, for example `~/github-token.txt`, so it never goes through your chat input or shell history, and ask the agent to store it. It calls the `set-environment-vars` tool for you: Prompt ``` Read the token in ~/github-token.txt and store it as GITHUB_TOKEN in my workspace. ``` The tool keeps your other variables and only adds or replaces this key; the same tool with a `null` value deletes it. Two things to know: the agent reads the file and sends the value in its tool call, so the token still lands in that session's transcript, and the tool's reply echoes the workspace's variables back to the agent, values included. Use it from a session you'd trust with every key in that workspace, and delete the file afterwards. `check-secrets` ([Step 4](#step-4)) then confirms the key is set. **API.** Send a `PATCH` request to the workspace's environment endpoint. The body must be wrapped in `environment_variables`. The call merges this key into your existing variables. Shell ``` curl -X PATCH "https://flows-api.jetty.io/api/v1/collections//environment" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"environment_variables": {"GITHUB_TOKEN": "github_pat_..."}}' ``` ### Step 4: Declare the secret in your runbook Important Workspace variables aren't automatically visible to a runbook. A runbook only receives the environment variables listed in the `secrets:` block of its frontmatter. If you skip this, the sandbox sees no `GITHUB_TOKEN` and the run stops at preflight. RUNBOOK.md frontmatter ``` --- name: github-haiku version: 1.0.0 description: Write an original haiku, commit it to a GitHub repository on a new branch and open a pull request secrets: GITHUB_TOKEN: { env: GITHUB_TOKEN } --- ``` `env:` names the environment variable the run delivers the value in; when the two names are the same, `GITHUB_TOKEN: {}` means the same thing. If you work through the Jetty plugin or MCP server, its `check-secrets` preflight reads this block and tells you which of the declared secrets your workspace has set. ## Create and test the runbook ### Step 5: Hand the token to git without writing it down One line makes every git command in the run authenticate: Shell ``` git config --global credential.helper '!f() { echo "username=x-access-token"; echo "password=${GITHUB_TOKEN}"; }; f' ``` Whenever git talks to `github.com`, it asks this helper for credentials, and the helper answers from the environment. The file git writes, `~/.gitconfig`, holds the text `${GITHUB_TOKEN}`, not the value. Don't put the token in the clone URL instead (`https://x-access-token:@github.com/...`): git saves that URL in `.git/config`, where every later command, log line and evaluation can read it. For the REST API, send the token as a header: `Authorization: Bearer $GITHUB_TOKEN`. ### Step 6: Create the task The example runbook has six steps. One of them is the work; the other five are the same for any runbook that lands its work in a repository: | Runbook step | What it does | When you adapt it | | --- | --- | --- | | 1\. Preflight | Checks that `GITHUB_TOKEN` is in the environment and git is installed; stops early if not | Keep | | 2\. Write the haiku | Produces the content | Replace with your own work | | 3\. Clone, commit and push | Sets the credential helper, clones the base branch, commits on a new branch (or the base branch) and pushes | Keep; make your changes after the `git clone` line | | 4\. Open the pull request | Posts the branch to the GitHub REST API; skipped on a direct push | Keep; change the title and body | | 5\. Verify | Confirms the branch is on the remote and the pull request is open | Keep | | 6\. Summary | Writes `summary.md` with `PASS` or `FAIL` and the reason | Keep | RUNBOOK.md ``` --- name: github-haiku version: 1.0.0 description: Write an original haiku, commit it to a GitHub repository as haikus/.md on a new branch and open a pull request, or push it straight to the base branch secrets: GITHUB_TOKEN: { env: GITHUB_TOKEN } --- # GitHub Haiku You write one original haiku, commit it to a GitHub repository as `haikus/.md` (for example `haikus/1792612345.md`) on a new branch, and open a pull request for it. When asked, you push the commit straight to the base branch instead of opening a pull request. ## Inputs - Repository, as `owner/name`: `{{repo}}` - Base branch: `{{base_branch}}` - Open a pull request (`true`) or push straight to the base branch (`false`): `{{open_pr}}` - Theme (optional, may be empty): `{{theme}}` The token is already in the environment as `GITHUB_TOKEN`. Never print, echo, or write its value to any file or log, never put it in a URL, and never run `env` or `printenv` without filtering it out. ## Step 1 — Preflight mkdir -p /app/results [ -n "${GITHUB_TOKEN:-}" ] || echo "MISSING GITHUB_TOKEN" git --version command -v gh >/dev/null && gh --version | head -1 || echo "gh: not installed (fine, this runbook uses curl)" If the token is missing, write `/app/results/summary.md` saying so, and stop. ## Step 2 — Write the haiku Compose one original haiku (three lines, 5-7-5 syllables). If a theme was given, use it; otherwise pick a seasonal image fitting today's date. Count the syllables of every line and rewrite until each line is exactly 5, 7, 5. Save it to `/app/results/haiku.txt` as exactly three lines followed by a trailing newline, with no title or other text. ## Step 3 — Clone, commit and push Run this exactly, as one script. It tells git to read the token from the environment (so the token is never written to disk), clones the base branch, adds the haiku on a new branch, and pushes. set -euo pipefail export GIT_TERMINAL_PROMPT=0 git config --global credential.helper '!f() { echo "username=x-access-token"; echo "password=${GITHUB_TOKEN}"; }; f' LOGIN=$(curl -sS -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github+json" \ https://api.github.com/user | python3 -c 'import sys,json; print(json.load(sys.stdin).get("login",""))') [ -n "$LOGIN" ] || { echo "GitHub rejected the token"; exit 1; } git config --global user.name "Jetty Runbook" git config --global user.email "$LOGIN@users.noreply.github.com" TS=$(date -u +%s) if [ "{{open_pr}}" = "false" ]; then BRANCH="{{base_branch}}"; else BRANCH="jetty/haiku-$TS"; fi rm -rf /app/work git clone -q --depth 1 --branch "{{base_branch}}" "https://github.com/{{repo}}.git" /app/work cd /app/work [ "$BRANCH" = "{{base_branch}}" ] || git checkout -q -b "$BRANCH" mkdir -p haikus { printf '# Haiku %s\n\n' "$TS"; cat /app/results/haiku.txt; } > "haikus/$TS.md" git add "haikus/$TS.md" git commit -q -m "Add haiku $TS" -m "Written by the github-haiku Jetty runbook." git push -q origin "HEAD:$BRANCH" printf '%s\n%s\n%s\n' "$TS" "$BRANCH" "$(git rev-parse HEAD)" > /app/results/commit.txt git ls-remote --heads origin "$BRANCH" | tee /app/results/remote_ref.txt `/app/results/remote_ref.txt` must show the commit hash from the third line of `commit.txt` next to `refs/heads/`. If the push was rejected, record the error message (never the token) in `/app/results/summary.md` and stop. ## Step 4 — Open the pull request Skip this step when `{{open_pr}}` is `false`. Otherwise run this exactly: set -euo pipefail TS=$(sed -n 1p /app/results/commit.txt); BRANCH=$(sed -n 2p /app/results/commit.txt) python3 - "$TS" "$BRANCH" <<'PY' > /tmp/pr_request.json import json, sys ts, branch = sys.argv[1], sys.argv[2] lines = open('/app/results/haiku.txt').read().strip().splitlines() body = "A haiku written by the github-haiku Jetty runbook.\n\n" + "\n".join("> " + l for l in lines) + "\n" print(json.dumps({"title": f"Add haiku {ts}", "head": branch, "base": "{{base_branch}}", "body": body})) PY curl -sS -X POST -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github+json" \ -H "X-GitHub-Api-Version: 2022-11-28" --data-binary @/tmp/pr_request.json \ "https://api.github.com/repos/{{repo}}/pulls" | tee /app/results/pr.json ## Step 5 — Verify For a pull request: `/app/results/pr.json` must contain a `number`, an `html_url`, `"state": "open"`, and a `head.ref` equal to the branch on the second line of `commit.txt`. Then confirm with GitHub: N=$(python3 -c 'import json; print(json.load(open("/app/results/pr.json")).get("number",""))') curl -sS -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github+json" \ "https://api.github.com/repos/{{repo}}/pulls/$N" \ | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get("state"), d.get("head",{}).get("ref"), d.get("html_url"))' For a direct push: `remote_ref.txt` already shows the commit on `{{base_branch}}`, and nothing more is needed. If `pr.json` has a `message` instead of a `number` (an error), retry Step 4 once. If the retry also fails, record the message (never the token) and stop. ## Step 6 — Summary Write `/app/results/summary.md` with: - the haiku - the repository, branch and commit hash, and the pull request URL (or "pushed to ``" when no PR was opened) - whether `gh` was available in the sandbox (from Step 1) - the result: `PASS` if Step 5 verified, otherwise `FAIL` with the reason ``` **Adapting the example.** Keep runbook Steps 1, 3, 4 and 5 as they are and replace Step 2 with your own work. For changes to the repository's own files, do the work after the `git clone` line, inside `/app/work`, and replace the `mkdir`, `printf` and `git add` lines with `git add -A`. Run the project's tests before the `git push` line so the pull request arrives green, and change the title and body in runbook Step 4 to describe the change. **Task configuration.** Create the task with default values for the variables, so a run with no inputs still knows where to push: JSON ``` "init_params": { "instruction": "", "vars": { "repo": "", "base_branch": "main", "open_pr": "true", "theme": "" } } ``` Use the `python312-uv` sandbox snapshot or similar; the runbook only needs `git`, `curl` and `python3`. This runbook finishes in a couple of minutes, so a `timeout_sec` of 900 is plenty. ### Step 7: Run a test Start a run from the Jetty web app, or from the API: Shell ``` curl -X POST "https://flows-api.jetty.io/api/v1/run//github-haiku" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -F 'init_params={"vars":{"repo":"","base_branch":"main","open_pr":"true","theme":"autumn"}}' ``` When the run finishes, its results include `pr.json` with the new pull request's `number` and `html_url`, `commit.txt` with the branch and commit hash, and `summary.md` showing `PASS`. In the repository you'll find a branch `jetty/haiku-` and an open pull request, for example: On GitHub ``` Add haiku 1792614032 #1 jetty/haiku-1792614032 → main ───────────────────────────────────────────────────────── A haiku written by the github-haiku Jetty runbook. > Cold wind strips the oak > leaves spiral down in silence > autumn holds its breath ``` To test the direct-push mode, pass `"open_pr":"false"`: the commit lands on `main` with no pull request. If `main` is protected, GitHub rejects the push with `GH006: Protected branch update failed` and the run records a `FAIL`. That's the protection doing its job, not a bug in the runbook. ## Running on a schedule The runbook works unchanged on a Jetty schedule: - **Inputs.** Scheduled runs pass `vars` (`repo`, `base_branch`, `open_pr`, `theme`) into the runbook's `{{…}}` variables. If a schedule doesn't set them, the task's default `vars` apply. - **No collisions.** Each run names its branch and file with the current epoch second, so runs never overwrite each other. - **Concurrency.** A personal access token doesn't rotate when used, so overlapping runs are fine. - **One pull request per run.** A daily schedule opens a daily pull request. Merge or close them, or change runbook Step 3 to reuse one fixed branch (`git push --force-with-lease`) and runbook Step 4 to update the existing pull request instead of opening another, if you'd rather have one living pull request than a pile. Before relying on a schedule, check two things: 1. **Token lifetime.** The token expires on the date you chose in [Step 2](#step-2), 366 days at most. After that, every run fails with `401 Bad credentials` until you store a new one ([Step 3](#step-3)), which an agent with the Jetty MCP server can do for you. 2. **Watch the result, not just the run status.** A run can show as `completed` even when the runbook recorded a `FAIL` in `summary.md`, for example because GitHub rejected the push. For unattended schedules, check `summary.md` or the repository's pull requests, or have your runbook send a notification on failure. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `summary.md` says `MISSING GITHUB_TOKEN` | The variable isn't listed in the runbook's `secrets:` frontmatter, or isn't set on the workspace | Add the `secrets:` block ([Step 4](#step-4)) and check the workspace environment ([Step 3](#step-3)) | | `GitHub rejected the token`, or `401 Bad credentials` | The token expired, was revoked, or was pasted with a typo | Create a new token ([Step 2](#step-2)) and update `GITHUB_TOKEN` ([Step 3](#step-3)), from the web app, your coding agent or the API | | `remote: Repository not found`, or the API returns `404 Not Found` | Wrong `repo` value, the token wasn't granted this repository, or the organization hasn't approved the token yet | Check `owner/name`; edit the token's **Repository access**; ask an organization owner to approve it | | `403 … Resource not accessible by personal access token` when opening the pull request | The token's **Pull requests** permission is missing or read-only | Edit the token: **Pull requests > Read and write** | | `remote: Permission to denied` or `403` on `git push` | The token's **Contents** permission is read-only | Edit the token: **Contents > Read and write** | | `GH006: Protected branch update failed` or `GH013: Repository rule violations found` on `git push` | The run pushed straight to a protected branch (`open_pr` is `false`) | Leave `open_pr` at `true`, or push to a branch the rules allow | | `refusing to allow a Personal Access Token to create or update workflow` | The change touches `.github/workflows/` | Add the **Workflows: Read and write** permission to the token | | `fatal: could not read Username for 'https://github.com'` | The credential helper wasn't configured, usually because the runbook's Step 3 script was run line by line in separate shells | Run the whole Step 3 block as one script | | `fatal: Remote branch main not found` | The repository is empty, or `base_branch` is wrong | Add a first commit ([Step 1](#step-1)) or fix `base_branch` | | `422 … A pull request already exists` or `No commits between main and ` | A branch name was reused, or the commit changed nothing | Keep one branch per run (the epoch suffix does this); check that runbook Step 3 wrote a file before committing | ## Security checklist - Keep the token to one repository and two permissions. Widen it only for what a runbook actually needs (Workflows for Actions files, Issues for issue comments). - Keep the token only in the Jetty workspace environment. Never put it in the runbook text or the task's `vars`. - Let git read the token through the credential helper ([Step 5](#step-5)), never through the clone URL. The runbook tells the agent never to print the token; keep that line if you adapt it. - Protect the base branch, so the runbook proposes and people merge. - Pull requests opened with the token appear under the token owner's name. If they should come from a bot identity, mint the token from a machine user account, or use a GitHub App. - To cut off access immediately, delete the token at [github.com/settings/personal-access-tokens](https://github.com/settings/personal-access-tokens) (or, for an organization, under its **Settings > Personal access tokens**). A run in progress fails at its next GitHub call. --- Source: https://jetty.io/docs/integrations/google-drive # Connecting a Jetty Runbook to Google Drive This guide explains how to give a Jetty runbook access to a folder in Google Drive, so that the runbook can write files to the folder and, optionally, read files from it. The worked example is a runbook that writes a short haiku to a text file named with the current UTC time in epoch seconds (for example `1792614032.txt`). The same setup applies to any runbook that saves reports, exports or other outputs to Drive, or that processes files placed in a Drive folder. **Time needed:** about 20 minutes, one time. **Prerequisites:** - A Google account that owns, or can edit, the destination Drive folder - Access to a Google Cloud project, or permission to create one - A Jetty workspace in which you can set environment variables and create tasks - `curl` and `python3` on your machine, for the one-time token step ## How it works Runbooks run in an isolated sandbox and cannot use the Google sign-in in your browser. Instead, the runbook authenticates with three credentials that you store in the Jetty workspace environment: | Variable | Description | | --- | --- | | `GDRIVE_CLIENT_ID` | The ID of an OAuth client that you create in Google Cloud | | `GDRIVE_CLIENT_SECRET` | The secret of that OAuth client | | `GDRIVE_REFRESH_TOKEN` | A long-lived token obtained by approving the client once in your browser | During each run, the runbook exchanges the refresh token for a short-lived access token and calls the Google Drive API with it. ### Scopes The refresh token carries one or more OAuth scopes, which are fixed when the token is minted ([Step 4](#step-4)). Decide which scopes the runbook needs before you begin, because changing them later means repeating Step 4. | Scope | Grants | Use it when | | --- | --- | --- | | `drive.file` | Create files, and read or modify the files that the runbook itself created. No access to anything else in the Drive. | The runbook only writes files, or reads back its own output. The example runbook uses this scope. | | `drive.readonly` | Read every file that the approving account can see, in any folder. | The runbook reads files that people or other applications placed in the folder. See [Reading files from Google Drive](#reading-files). | | `drive` | Full read and write access to every file that the approving account can see. | The runbook has to modify files that it did not create. | The `drive.file` scope is sufficient to upload into a folder that the approving account created in the Drive web UI. The full `drive` scope is not required for writing. > **Warning:** A `drive.readonly` or `drive` token can read everything the approving account can see, not just the target folder. Mint such a token from an account that only has access to the folders the runbook needs, for example a dedicated account with which the folder is shared. > **Note:** Google service accounts are not used in this guide because they have no storage quota of their own. Uploads from a service account into a personal _My Drive_ folder fail with `storageQuotaExceeded`. Service accounts work only with Shared Drives. If the destination is a Shared Drive, a service account is a valid alternative. This guide covers the more common My Drive case. ## Set up Google Cloud Steps 1 to 4 are performed once, in the Google Cloud console and on your own machine. ### Step 1: Enable the Google Drive API 1. Open the [Google Cloud console](https://console.cloud.google.com/) and select a project, or create one. 2. Go to **APIs & Services > Library**, search for **Google Drive API** and click **Enable**. If the API is not enabled, uploads fail with `403 … Google Drive API has not been used in project … or it is disabled`. ### Step 2: Configure the OAuth consent screen Open **Google Auth Platform** (or **APIs & Services > OAuth consent screen**) and configure the following settings: 1. **Audience (user type).** Select **Internal** if you use Google Workspace and only people in your organization will authorize the app. Internal apps have no token expiry issues and are recommended. Otherwise, select **External**. 2. **Data access (scopes).** Add `https://www.googleapis.com/auth/drive.file`. If the runbook reads files that it did not create, also add `https://www.googleapis.com/auth/drive.readonly` (see [Scopes](#scopes)). 3. **Publishing status (External apps only).** Click **Publish app** so that the status changes to **In production**. > **Important:** The publishing status determines whether scheduled runs keep working. While an External app is in **Testing** status, Google expires its refresh tokens after 7 days: the runbook works for a week and then fails with `invalid_grant`. In production, the token remains valid until you revoke it, or until it goes unused for 6 months. > **Note:** Google does not require a verification review for `drive.file`. `drive.readonly` is a restricted scope, but nothing changes for an Internal app, and for an External app that you created for yourself a review is needed only to remove the "Google hasn't verified this app" screen or to let more than 100 users authorize the app. When you approve the app in Step 4, you may see that screen. It is expected for an app you created yourself: click **Advanced > Continue**. ### Step 3: Create an OAuth client 1. Go to **APIs & Services > Credentials > Create credentials > OAuth client ID**. 2. For **Application type**, select **Desktop app** and enter a name, for example `jetty-drive-runbook`. 3. Copy the **Client ID** and the **Client secret**. ### Step 4: Mint a refresh token Save the following script as `mint-drive-token.sh`. It opens your browser for a one-time approval, then exchanges the resulting authorization code for a refresh token. Shell ``` #!/usr/bin/env bash # Mints a Google Drive refresh token (drive.file scope) for a Desktop OAuth client. set -euo pipefail : "${GDRIVE_CLIENT_ID:?set GDRIVE_CLIENT_ID}" "${GDRIVE_CLIENT_SECRET:?set GDRIVE_CLIENT_SECRET}" PORT=8765 REDIRECT="http://localhost:$PORT/callback" SCOPE="https://www.googleapis.com/auth/drive.file" URL="https://accounts.google.com/o/oauth2/v2/auth?client_id=$GDRIVE_CLIENT_ID&redirect_uri=$REDIRECT&response_type=code&access_type=offline&prompt=consent&scope=$SCOPE" echo "Opening your browser to approve Drive access..." ( open "$URL" 2>/dev/null || xdg-open "$URL" 2>/dev/null || echo "Open this URL: $URL" ) & # Wait for Google to redirect back with ?code=... CODE=$(python3 - "$PORT" <<'PY' import http.server, sys, urllib.parse class H(http.server.BaseHTTPRequestHandler): code = "" def do_GET(self): H.code = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query).get("code", [""])[0] self.send_response(200); self.end_headers() self.wfile.write(b"Done - you can close this tab.") def log_message(self, *a): pass http.server.HTTPServer(("localhost", int(sys.argv[1])), H).handle_request() print(H.code) PY ) # Exchange the one-time code for a refresh token curl -s https://oauth2.googleapis.com/token \ --data-urlencode "client_id=$GDRIVE_CLIENT_ID" \ --data-urlencode "client_secret=$GDRIVE_CLIENT_SECRET" \ --data-urlencode "code=$CODE" \ --data-urlencode "redirect_uri=$REDIRECT" \ -d grant_type=authorization_code \ | python3 -c 'import sys,json; d=json.load(sys.stdin); print("GDRIVE_REFRESH_TOKEN=" + d["refresh_token"]) if "refresh_token" in d else sys.exit("No refresh token returned: %s" % d)' ``` The script requests the `drive.file` scope. To request `drive.readonly` as well, change the `SCOPE` line as follows (`%20` separates the two scopes): Shell ``` SCOPE="https://www.googleapis.com/auth/drive.file%20https://www.googleapis.com/auth/drive.readonly" ``` Run the script on your own machine with the client ID and secret from Step 3: Shell ``` chmod +x mint-drive-token.sh GDRIVE_CLIENT_ID=... GDRIVE_CLIENT_SECRET=... ./mint-drive-token.sh ``` When the browser opens, sign in as the Google account that owns the destination folder and approve access. The script prints `GDRIVE_REFRESH_TOKEN=...`. Handle this value as you would a password. ## Configure Jetty ### Step 5: Store the credentials in the workspace environment Add the three values as environment variables in your Jetty workspace, using either the web app or the API. **Web app.** Open **Settings > Integrations** and add `GDRIVE_CLIENT_ID`, `GDRIVE_CLIENT_SECRET` and `GDRIVE_REFRESH_TOKEN`. ![The Integrations page under Settings in the Jetty web app, listing GDRIVE\_CLIENT\_ID, GDRIVE\_CLIENT\_SECRET and GDRIVE\_REFRESH\_TOKEN with their values masked and edit and delete actions on each row.](/_next/image?url=%2Fintegrations%2Fgoogle-drive%2Fsettings-integrations.png&w=3840&q=75) Settings > Integrations after Step 5: the three variables, with their values masked. **API.** Send a `PATCH` request to the workspace's environment endpoint. The body must be wrapped in `environment_variables`. The call merges these keys into the existing variables. Shell ``` curl -X PATCH "https://flows-api.jetty.io/api/v1/collections//environment" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"environment_variables": { "GDRIVE_CLIENT_ID": "...", "GDRIVE_CLIENT_SECRET": "...", "GDRIVE_REFRESH_TOKEN": "..." }}' ``` ### Step 6: Declare the secrets in the runbook frontmatter > **Important:** A runbook receives only the environment variables that are listed in the `secrets:` block of its frontmatter. Workspace variables are not visible to the runbook otherwise. If the block is missing, the sandbox sees none of the three variables. YAML ``` --- name: drive-haiku version: 1.0.1 description: Write an original haiku and upload it to a Google Drive folder as .txt secrets: GDRIVE_CLIENT_ID: { env: GDRIVE_CLIENT_ID, required: true } GDRIVE_CLIENT_SECRET: { env: GDRIVE_CLIENT_SECRET, required: true } GDRIVE_REFRESH_TOKEN: { env: GDRIVE_REFRESH_TOKEN, required: true } --- ``` ### Step 7: Find the folder ID Open the destination folder in Google Drive. The folder ID is the last segment of the URL: Text ``` https://drive.google.com/drive/folders/1Y_Cm0q8zWHQEN6LM_QSO1xGY_cGwgliO └──────────── folder ID ────────────┘ ``` Pass the folder ID to the runbook as a variable rather than hard-coding it. The same runbook can then write to different folders, and a schedule can supply the folder at run time. ## Create and test the runbook ### Step 8: Create the task The complete example runbook follows. Its Steps 1, 3 and 4 (preflight, upload and verify) are generic; Step 2 produces the content. Markdown ``` --- name: drive-haiku version: 1.0.1 description: Write an original haiku and upload it to a Google Drive folder as .txt secrets: GDRIVE_CLIENT_ID: { env: GDRIVE_CLIENT_ID, required: true } GDRIVE_CLIENT_SECRET: { env: GDRIVE_CLIENT_SECRET, required: true } GDRIVE_REFRESH_TOKEN: { env: GDRIVE_REFRESH_TOKEN, required: true } --- # Drive Haiku You write one original haiku and upload it as a plain-text file to a Google Drive folder. The file name is the current UTC time in Unix epoch seconds, e.g. `1792612345.txt`. ## Inputs - Drive folder id: `{{folder_id}}` - Theme (optional, may be empty): `{{theme}}` Credentials are already in the environment: `GDRIVE_CLIENT_ID`, `GDRIVE_CLIENT_SECRET`, `GDRIVE_REFRESH_TOKEN`. Never print, echo, or write their values to any file or log. ## Step 1 — Preflight mkdir -p /app/results for v in GDRIVE_CLIENT_ID GDRIVE_CLIENT_SECRET GDRIVE_REFRESH_TOKEN; do [ -n "${!v:-}" ] || echo "MISSING $v" done If any variable is missing, write `/app/results/summary.md` saying which one, and stop. ## Step 2 — Write the haiku Compose one original haiku (three lines, 5-7-5 syllables). If a theme was given, use it; otherwise pick a seasonal image fitting today's date. Count the syllables of every line and rewrite until each line is exactly 5, 7, 5. Save it to `/app/results/haiku.txt` as exactly three lines followed by a trailing newline, with no title or other text. ## Step 3 — Upload Run this exactly (it fetches an access token, then does a multipart upload into the folder): set -euo pipefail TS=$(date -u +%s) NAME="$TS.txt" cp /app/results/haiku.txt "/app/results/$NAME" AT=$(curl -sS https://oauth2.googleapis.com/token \ --data-urlencode "client_id=$GDRIVE_CLIENT_ID" \ --data-urlencode "client_secret=$GDRIVE_CLIENT_SECRET" \ --data-urlencode "refresh_token=$GDRIVE_REFRESH_TOKEN" \ -d grant_type=refresh_token | python3 -c 'import sys,json; print(json.load(sys.stdin).get("access_token",""))') [ -n "$AT" ] || { echo "token exchange failed"; exit 1; } printf '{"name":"%s","mimeType":"text/plain","parents":["%s"]}' "$NAME" "{{folder_id}}" > /tmp/meta.json curl -sS -X POST -H "Authorization: Bearer $AT" \ -F "metadata=.txt`, and `parents` containing `{{folder_id}}`. If it has an `error` instead, retry Step 3 once. If the retry also fails, record the error message (never credentials) and stop. ## Step 5 — Summary Write `/app/results/summary.md` with: - the haiku - the file name and Drive file id/link - the result: `PASS` if Step 4 verified, otherwise `FAIL` with the reason ``` **Adapting the example.** Keep runbook Steps 1, 3 and 4 unchanged and replace Step 2 with your own work. Point the `cp` command at the file you produce, and set `mimeType` and the file's `type=` to match, for example `text/csv` for a CSV file or `application/pdf` for a PDF. **Task configuration.** Create the task with default values for its variables, so that a run with no inputs still knows where to write: JSON ``` "init_params": { "instruction": "", "vars": { "folder_id": "", "theme": "" } } ``` Use the `python312-uv` sandbox snapshot or a similar one; the runbook needs only `curl` and `python3`. The runbook finishes in a few minutes, so a `timeout_sec` of 900 is sufficient. ### Step 9: Run a test Start a run from the Jetty web app, or with the API: Shell ``` curl -X POST "https://flows-api.jetty.io/api/v1/run//drive-haiku" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -F 'init_params={"vars":{"folder_id":"","theme":"autumn"}}' ``` When the run finishes, its results include `drive_response.json` with the new file's `id` and `webViewLink`, and `summary.md` showing `PASS`. The file also appears in the Drive folder, for example: Text ``` 1792614032.txt ───────────────────────────── Cold wind strips the oak leaves spiral down in silence autumn holds its breath ``` ## Reading files from Google Drive A runbook can also read from Drive, for example to pick up a CSV file that a colleague placed in the folder, or to summarize a Google Doc. The credentials, the workspace variables and the `secrets:` block are the same as for writing. Two things differ: the token must carry the `drive.readonly` scope, and the runbook calls different API endpoints. ### Requirements The `drive.file` scope covers only files that the runbook created itself. Files that a person placed in the folder are invisible to it: a folder listing omits them, and fetching one by ID returns `404 File not found`, even though the account that approved the token can open the file in the browser. In a folder holding one runbook-written file and one hand-uploaded file, a `drive.file` token lists only the first. To read files that it did not create, the runbook needs the `https://www.googleapis.com/auth/drive.readonly` scope. If the current token has only `drive.file`: 1. Add `drive.readonly` to the consent screen scopes ([Step 2](#step-2)). 2. Mint a new token with the two-scope `SCOPE` line ([Step 4](#step-4)). The client ID and secret do not change. 3. Store the new `GDRIVE_REFRESH_TOKEN` in the workspace ([Step 5](#step-5)). If the runbook only reads and never writes, request `drive.readonly` alone. The publishing rule from Step 2 still applies: keep the app In production, or the token expires after 7 days. For the access that `drive.readonly` grants, see [Scopes](#scopes). ### List the files in a folder Obtain an access token exactly as in Step 3 of the example runbook, then list the folder, newest first: Shell ``` curl -sS -G -H "Authorization: Bearer $AT" "https://www.googleapis.com/drive/v3/files" \ --data-urlencode "q='{{folder_id}}' in parents and trashed=false" \ --data-urlencode "fields=files(id,name,mimeType,modifiedTime)" \ --data-urlencode "orderBy=modifiedTime desc" \ --data-urlencode "pageSize=100" ``` To find a single file by name, add `and name='report.csv'` to the `q` value. If the folder is in a Shared Drive, also pass `supportsAllDrives=true` and `includeItemsFromAllDrives=true`. ### Download a file Files that were uploaded as files (`text/csv`, `application/pdf`, images and so on) are downloaded as they are: Shell ``` curl -sS -L -H "Authorization: Bearer $AT" \ "https://www.googleapis.com/drive/v3/files/$FILE_ID?alt=media" -o /app/inputs/report.csv ``` ### Export a Google Docs, Sheets or Slides file Google Docs, Sheets and Slides files have the `mimeType` `application/vnd.google-apps.document`, `.spreadsheet` or `.presentation`. They have no binary content to download, so `alt=media` fails with `403 Only files with binary content can be downloaded. Use Export with Docs Editors files.` Export them to a plain format instead: Shell ``` curl -sS -H "Authorization: Bearer $AT" \ "https://www.googleapis.com/drive/v3/files/$FILE_ID/export?mimeType=text/markdown" -o /app/inputs/brief.md ``` Useful export types: `text/markdown` or `text/plain` for a Doc, `text/csv` for the first sheet of a spreadsheet, and `application/pdf` for any of them. Exports are limited to 10 MB. ### Example: a fetch step for the runbook Keep Steps 1, 3 and 4 of the example runbook and replace Step 2 with a fetch step. The following step downloads a file named by a new `{{input_name}}` variable into `/app/inputs/`, where the rest of the runbook can process it: Markdown ``` ## Step 2 — Fetch the input Run this exactly. It downloads `{{input_name}}` from the Drive folder into `/app/inputs/`: set -euo pipefail mkdir -p /app/inputs AT=$(curl -sS https://oauth2.googleapis.com/token \ --data-urlencode "client_id=$GDRIVE_CLIENT_ID" \ --data-urlencode "client_secret=$GDRIVE_CLIENT_SECRET" \ --data-urlencode "refresh_token=$GDRIVE_REFRESH_TOKEN" \ -d grant_type=refresh_token | python3 -c 'import sys,json; print(json.load(sys.stdin).get("access_token",""))') [ -n "$AT" ] || { echo "token exchange failed"; exit 1; } FILE_ID=$(curl -sS -G -H "Authorization: Bearer $AT" "https://www.googleapis.com/drive/v3/files" \ --data-urlencode "q='{{folder_id}}' in parents and name='{{input_name}}' and trashed=false" \ --data-urlencode "orderBy=modifiedTime desc" --data-urlencode "fields=files(id,mimeType)" \ | python3 -c 'import sys,json; f=json.load(sys.stdin).get("files",[]); print(f[0]["id"] if f else "")') [ -n "$FILE_ID" ] || { echo "no file named {{input_name}} in the folder"; exit 1; } curl -sS -L -H "Authorization: Bearer $AT" \ "https://www.googleapis.com/drive/v3/files/$FILE_ID?alt=media" -o "/app/inputs/{{input_name}}" wc -c "/app/inputs/{{input_name}}" If the download fails, record the error message (never credentials) in `/app/results/summary.md` and stop. ``` Add `input_name` to the task's default `vars` next to `folder_id`. Because the token carries both scopes, the same run can read its input and upload its result with the unchanged Steps 3 and 4. As with the upload step, give the agent the exact `curl` commands rather than an instruction such as "download the file from Drive". Left to improvise, a model tends to reach for `gdown`, the Python Drive client or a public share link, which are either missing from the sandbox or require the file to be public. ## Running on a schedule The runbook works unchanged on a Jetty schedule: - **Inputs.** Scheduled runs pass `vars` (`folder_id`, `theme`) into the runbook's `{{…}}` variables. If a schedule does not set them, the task's default `vars` apply. - **File names.** Each run names its file with the current epoch second, so runs never overwrite each other. - **Concurrency.** Google refresh tokens do not rotate when used, so overlapping runs are safe. Before relying on a schedule, verify two things: 1. **Token lifetime.** The OAuth app must be **Internal** or **In production** ([Step 2](#step-2)). In Testing status, the schedule stops working after 7 days. 2. **Run results.** A run can finish as _completed_ even when the runbook recorded `FAIL` in `summary.md`, for example because Google rejected the upload. For unattended schedules, check `summary.md` or the folder contents, or have the runbook send a notification on failure. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `summary.md` says `MISSING GDRIVE_…` | The variables are not listed in the runbook's `secrets:` frontmatter, or are not set in the workspace | Add the `secrets:` block ([Step 6](#step-6)) and check the workspace environment ([Step 5](#step-5)) | | `403 … Google Drive API has not been used in project …` | The Drive API is not enabled in the OAuth client's project | Enable it ([Step 1](#step-1)). Allow a few minutes for the change to take effect | | `invalid_grant` during token exchange | The refresh token expired (Testing status, 7 days), was revoked, or the account password changed | Publish the app ([Step 2](#step-2)), mint a new token ([Step 4](#step-4)) and update `GDRIVE_REFRESH_TOKEN` ([Step 5](#step-5)) | | `404 File not found: ` | Wrong folder ID, or the token belongs to a Google account that cannot access the folder | Check the ID ([Step 7](#step-7)), and approve access in Step 4 as an account with edit access to the folder | | `storageQuotaExceeded` | A service account is being used with a My Drive folder | Use the OAuth refresh-token setup in this guide, or a Shared Drive | | Step 4 printed no `refresh_token` | The client was approved before and Google did not issue a new token | The script forces a new token with `prompt=consent`. If it still fails, remove the app's access at [myaccount.google.com/permissions](https://myaccount.google.com/permissions) and run Step 4 again | | `404 File not found: ` for a file that is visible in Drive | The token has only `drive.file`, and the runbook did not create that file | Mint a new token that includes `drive.readonly` (see [Reading files from Google Drive](#reading-files)) | | `403 Only files with binary content can be downloaded` | The file is a Google Docs, Sheets or Slides file | Use the `export` endpoint with a `mimeType` instead of `alt=media` | ## Security considerations - Use `drive.file` for runbooks that only write. Add `drive.readonly` only for runbooks that read files they did not create, and use the full `drive` scope only if a runbook has to modify such files. - A `drive.readonly` token can read everything the approving account can see, not just the folder. Mint it from an account that only has access to the folders the runbook needs. - Keep the credentials only in the Jetty workspace environment. Never put them in the runbook text or in the task's `vars`. - The example runbook instructs the agent never to print the credentials. Keep that instruction when you adapt it. - To revoke access immediately, remove the app at [myaccount.google.com/permissions](https://myaccount.google.com/permissions) or delete the OAuth client in Google Cloud. --- Source: https://jetty.io/docs/integrations/slack # Connecting a Jetty Runbook to Slack This guide shows how to let a Jetty runbook post messages to a Slack channel and, when it needs to, read the messages already there. The worked example is a runbook that writes a short haiku, posts it to a channel as a bot, and confirms that Slack stored it by fetching the message's permalink. You can reuse the same setup for any runbook that posts a digest, a report, an alert on a failing check, or anything else your team should see in Slack. **Time needed:** about 10 minutes, one time. **Prerequisites:** - A Slack workspace where you can create and install apps, or an admin who approves them - A channel for the runbook to post to, where you can invite the bot - A Jetty workspace where you can set environment variables and create tasks ## How it works Runbooks run in an isolated sandbox with no Slack login. You give the runbook one credential: | Variable | Description | | --- | --- | | `SLACK_BOT_TOKEN` | The bot token of a small Slack app you create for your runbooks (it starts with `xoxb-`), allowed to post messages and nothing else | During each run, the runbook sends the token as a Bearer header to the Slack Web API: `auth.test` to check it, `chat.postMessage` to post, and `chat.getPermalink` to confirm the message exists. `SLACK_BOT_TOKEN` is the name Jetty's [integrations](/docs/integrations) page uses for a Slack bot token. **Scope.** The app gets one bot scope, `chat:write`. It can post in channels the bot has been invited to, reply in threads and edit its own messages, and nothing else: no reading messages, no listing channels. `auth.test` and `chat.getPermalink` need no scope at all. A runbook that reads a channel also needs `channels:history` (see [Reading from Slack](#reading-from-slack)); scopes can be added later without changing the token. **Every reply is HTTP 200.** Slack reports failures in the body, as `"ok": false` with an `error` name, so the runbook checks `ok` after every call rather than the status code. > **Note:** Why not an incoming webhook, or a user token? An [incoming webhook](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks) is simpler, but it posts to the one channel chosen when it was created and answers with a plain `ok`: no message timestamp, so a runbook can't confirm the post, reply in its thread or edit it later. A user token (`xoxp-`) posts as a person and carries everything that person can see. A bot token with `chat:write` is the smallest credential that can post, confirm and follow up. ## Set up Slack Steps 1 to 3 are done once, in your browser and in Slack. ### Step 1: Create a Slack app The quickest way is from a manifest, which sets the app's name, its bot user and its scope in one paste. 1. Open [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App > From a manifest**. 2. Pick the workspace the runbook will post to and click **Next**. 3. Switch the editor to **YAML**, replace its contents with the manifest below, then click **Next > Create**. App manifest ``` display_information: name: Jetty Runbook Example description: Posts messages from Jetty runbooks features: bot_user: display_name: Jetty Runbook Example always_online: false oauth_config: scopes: bot: - chat:write settings: org_deploy_enabled: false socket_mode_enabled: false token_rotation_enabled: false ``` The two names are what your team sees next to the messages; change them as you like. The app has no commands or event subscriptions, so there is nothing to host. > **Warning:** Leave `token_rotation_enabled` at `false`. With [token rotation](https://docs.slack.dev/authentication/using-token-rotation) on, the bot token expires every 12 hours and a runbook has no way to refresh it, so scheduled runs fail within a day. Slack doesn't let you turn rotation off again once it's on. ### Step 2: Install the app and copy the token 1. Open **Install App** in the app's sidebar and click **Install to Workspace**, then **Allow**. If your workspace requires approval for new apps, an admin approves it first. 2. Copy the **Bot User OAuth Token** that appears. It starts with `xoxb-`. ### Step 3: Invite the bot to the channel In the channel the runbook will post to, send `/invite @Jetty Runbook Example` (the app's name from Step 1). Slack confirms that the app was added. A bot with only `chat:write` can post nowhere else, so this is how you decide, channel by channel, where it may write. ## Configure Jetty ### Step 4: Store the token in your Jetty workspace Add `SLACK_BOT_TOKEN` as an environment variable on your Jetty workspace. Any of these three ways works. **Web app.** Open **Settings > Integrations** and add `SLACK_BOT_TOKEN`. **Coding agent.** If you work through the Jetty plugin for Claude Code or the [Jetty MCP server](/docs/integrations/mcp-server) in Cursor, VS Code, Windsurf or Zed, save the token in a file of its own, for example `~/slack-token.txt`, so it never goes through your chat input or shell history, and ask the agent to store it. It calls the `set-environment-vars` tool for you: Prompt ``` Read the token in ~/slack-token.txt and store it as SLACK_BOT_TOKEN in my workspace. ``` The tool keeps your other variables and only adds or replaces this key. Two things to know: the agent reads the file and sends the value in its tool call, so the token still lands in that session's transcript, and the tool's reply echoes the Jetty workspace's variables back to the agent, values included. Use it from a session you'd trust with every key in that Jetty workspace, and delete the file afterwards. **API.** Send a `PATCH` to the Jetty workspace's environment endpoint. The body must be wrapped in `environment_variables`; the call merges this key into your existing variables. Shell ``` curl -X PATCH "https://flows-api.jetty.io/api/v1/collections//environment" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"environment_variables": {"SLACK_BOT_TOKEN": "xoxb-..."}}' ``` ### Step 5: Declare the secret in your runbook > **Important:** Jetty workspace variables aren't automatically visible to a runbook. A runbook only receives the environment variables listed in the `secrets:` block of its frontmatter. If you skip this, the sandbox sees no `SLACK_BOT_TOKEN` and the run stops at preflight. RUNBOOK.md frontmatter ``` --- name: slack-post version: 1.0.0 description: Write an original haiku and post it to a Slack channel through the Slack Web API, then confirm the message exists with its permalink secrets: SLACK_BOT_TOKEN: { env: SLACK_BOT_TOKEN } --- ``` `env:` names the environment variable the run delivers the value in. If you work through the Jetty plugin or MCP server, its `check-secrets` preflight reads this block and tells you which declared secrets your Jetty workspace has set. ### Step 6: Find the channel ID Click the channel's name at the top of the channel: the ID is at the bottom of the **About** tab, next to a copy button, and starts with `C` (or `G` for some private channels). In Slack in a browser, it's also the last part of the URL, `https://app.slack.com/client/TXXXXXXX/CXXXXXXX`. Pass the ID to the runbook as a variable rather than a channel name: an ID survives the channel being renamed, and a schedule can pick the channel at run time. ## Create and test the runbook ### Step 7: Create the task The complete example runbook follows. Its Steps 1, 3, 4 and 5 (preflight, post, verify and summary) are generic; Step 2 writes the message. RUNBOOK.md ```` --- name: slack-post version: 1.0.0 description: Write an original haiku and post it to a Slack channel through the Slack Web API, then confirm the message exists with its permalink secrets: SLACK_BOT_TOKEN: { env: SLACK_BOT_TOKEN } --- # Slack Post You write one original haiku and post it to a Slack channel as the bot whose token is in the environment, then confirm that Slack stored the message by fetching its permalink. ## Inputs - Channel ID (starts with `C` for a channel; the bot must be a member): `{{channel}}` - Theme (optional, may be empty): `{{theme}}` The token is already in the environment as `SLACK_BOT_TOKEN`. Never print, echo, or write its value to any file or log, never put it in a URL, and never run `env` or `printenv` without filtering it out. Run with the inputs exactly as given, even if one looks wrong. Do not ask anyone anything: if an input is wrong, Slack says so, and Step 4 records the error as a `FAIL`. Always finish with `/app/results/summary.md`. ## Step 1 — Preflight ```bash mkdir -p /app/results [ -n "${SLACK_BOT_TOKEN:-}" ] || echo "MISSING SLACK_BOT_TOKEN" curl --version | head -1; python3 --version ``` If the token is missing, write `/app/results/summary.md` saying so, and stop. ## Step 2 — Write the haiku Compose one original haiku (three lines, 5-7-5 syllables). If a theme was given, use it; otherwise pick a seasonal image fitting today's date. Count the syllables of every line and rewrite until each line is exactly 5, 7, 5. Save it to `/app/results/haiku.txt` as exactly three lines followed by a trailing newline, with no title or other text. ## Step 3 — Post the message Run this exactly, as one script. Slack answers every Web API call with HTTP 200, success or not: the JSON field `ok` says which, and `error` names the problem, so the script checks `ok` itself. ```bash set -euo pipefail cd /app/results curl -sS -X POST -H "Authorization: Bearer $SLACK_BOT_TOKEN" https://slack.com/api/auth.test > auth.json python3 -c 'import json; d=json.load(open("auth.json")); assert d.get("ok"), d.get("error"); print("token ok:", d["team"], d["user"])' python3 - <<'PY' > message.json import json haiku = open("/app/results/haiku.txt").read().strip() print(json.dumps({"channel": "{{channel}}", "text": haiku + "\n_Posted by the slack-post Jetty runbook._", "unfurl_links": False})) PY curl -sS -X POST -H "Authorization: Bearer $SLACK_BOT_TOKEN" -H "Content-Type: application/json; charset=utf-8" \ --data-binary @message.json https://slack.com/api/chat.postMessage > post.json python3 -c 'import json; d=json.load(open("post.json")); assert d.get("ok"), d.get("error"); print("posted", d["channel"], d["ts"])' TS=$(python3 -c 'import json; print(json.load(open("post.json"))["ts"])') curl -sS -G -H "Authorization: Bearer $SLACK_BOT_TOKEN" --data-urlencode "channel={{channel}}" \ --data-urlencode "message_ts=$TS" https://slack.com/api/chat.getPermalink > permalink.json python3 -c 'import json; d=json.load(open("permalink.json")); assert d.get("ok"), d.get("error"); print(d["permalink"])' ``` ## Step 4 — Verify `post.json` must have `"ok": true`, a `ts`, and a `channel` equal to `{{channel}}`; `permalink.json` must have `"ok": true` and a `permalink`. If a call returned `"ok": false`, its `error` names the cause (for example `invalid_auth`, `not_authed`, `channel_not_found`, `not_in_channel`, `missing_scope` or `ratelimited`). On `ratelimited`, wait the number of seconds Slack asks for and run Step 3 once more. Otherwise do not retry: record the call, the `error` value and any `needed` scope (never the token) in `/app/results/summary.md` and stop. ## Step 5 — Summary Write `/app/results/summary.md` with: - the haiku - the workspace and bot user from `auth.json`, the channel, the message `ts` and the permalink - the result: `PASS` if Step 4 verified, otherwise `FAIL` with the call and the `error` value ```` To reuse this for your own runbook, keep Steps 1, 3, 4 and 5 as they are, replace Step 2 with your own work that writes the message to a file, and point the `haiku = open(...)` line in Step 3 at that file. Keep the text under 4,000 characters (Slack truncates past 40,000) and link to anything longer. For a richer layout, add a `blocks` array next to `text`; `text` then becomes the fallback shown in notifications. **Task configuration.** Create the task with default values for the variables, so a run with no inputs still knows where to post: JSON ``` "init_params": { "instruction": "", "vars": { "channel": "", "theme": "" } } ``` Use the `python312-uv` sandbox snapshot or similar; the runbook only needs `curl` and `python3`. It finishes in under a minute, so a `timeout_sec` of 900 is plenty. ### Step 8: Run a test Start a run from the Jetty web app, or from the API: Shell ``` curl -X POST "https://flows-api.jetty.io/api/v1/run//slack-post" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -F 'init_params={"vars":{"channel":"","theme":"autumn"}}' ``` When the run finishes, its results include `post.json` with the message's `ts`, `permalink.json` with its link, and `summary.md` showing `PASS`. In the channel, the message appears under the app's name, for example: In the channel ``` Jetty Runbook Example APP Maple leaves let go Drifting down past cooling eaves Fall fills the still yard Posted by the slack-post Jetty runbook. ``` To see a failure, pass a channel ID the bot can't use, such as `C00000000`: the run records `FAIL` with `chat.postMessage` and `channel_not_found`, and nothing is posted. ## Reading from Slack A runbook can also read a channel, for example to summarise yesterday's messages or pick up a request someone posted there. This needs one more scope: open the app's **OAuth & Permissions > Scopes > Bot Token Scopes**, add `channels:history` (`groups:history` for a private channel) and reinstall the app when Slack asks. The token keeps its value, so nothing changes in Jetty. The bot reads only channels it has been invited to. Fetch the most recent messages, newest first: Shell ``` curl -sS -G -H "Authorization: Bearer $SLACK_BOT_TOKEN" "https://slack.com/api/conversations.history" \ --data-urlencode "channel={{channel}}" \ --data-urlencode "limit=200" > history.json ``` The reply is a `messages` list, each with its `ts` and `text`. - **More pages.** When `has_more` is `true`, send `response_metadata.next_cursor` back as `cursor`. - **One message.** To fetch a message whose `ts` you know, send `oldest=`, `inclusive=true` and `limit=1`. - **Threads.** History doesn't include replies inside threads. A parent message carries a `reply_count`; `conversations.replies` with `channel` and `ts=` returns the thread, under the same scope. As with posting, give the agent these exact `curl` commands rather than "read the channel": left to improvise, a model tends to reach for a Slack SDK the sandbox doesn't have. ## Running on a schedule The runbook works unchanged on a Jetty schedule: - **Inputs.** Scheduled runs pass `vars` (`channel`, `theme`) into the runbook's `{{…}}` variables. If a schedule doesn't set them, the task's default `vars` apply. - **Token lifetime.** A bot token without rotation doesn't expire. It stops working only when someone revokes it or removes the app. - **Concurrency.** The token doesn't rotate when used, so overlapping runs are fine. `chat.postMessage` allows about one message per second per channel; past that, Slack answers HTTP `429` with `ratelimited` and a `Retry-After` header. - **One message per run.** A daily schedule posts a daily message. To keep the channel quiet, have each run reply in one thread (`"thread_ts": ""` in the JSON body) or update one living message with `chat.update` and its `ts`. Both work with `chat:write` alone. Before relying on a schedule, check two things: 1. **Rotation is off.** `token_rotation_enabled: false` in the app's manifest ([Step 1](#step-1)). 2. **Watch the result, not just the run status.** A run can show as _completed_ even when the runbook recorded a `FAIL` in `summary.md`, for example because Slack answered `channel_not_found`. For unattended schedules, check `summary.md`, or have a later step alert you when it says `FAIL`. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `summary.md` says `MISSING SLACK_BOT_TOKEN` | The variable isn't listed in the runbook's `secrets:` frontmatter, or isn't set on the Jetty workspace | Add the `secrets:` block ([Step 5](#step-5)) and check the Jetty workspace environment ([Step 4](#step-4)) | | `not_authed` | No token reached Slack: the `Authorization` header is missing or the variable is empty | Same as above; the token goes in an `Authorization: Bearer` header | | `invalid_auth` | The token is wrong, or was sent as a `?token=` query parameter, which Slack rejects | Copy the token again ([Step 2](#step-2)) and update it ([Step 4](#step-4)); send it as a header | | `token_revoked` or `account_inactive` | The token was revoked, or the app was removed from the workspace | Reinstall the app ([Step 2](#step-2)) and store the new token ([Step 4](#step-4)) | | `channel_not_found` | The ID is wrong, or the bot can't see the channel. Posting by name to a channel the bot isn't in, such as `#general`, fails this way too | Copy the ID again ([Step 6](#step-6)) and invite the bot ([Step 3](#step-3)) | | `not_in_channel` | The bot hasn't been invited to the channel | Invite it ([Step 3](#step-3)), or add the `chat:write.public` scope to let it post in any public channel | | `missing_scope`, with `needed` and `provided` fields | The call needs a scope the token doesn't have; `needed` lists the ones that would do | Add one of them under **OAuth & Permissions > Scopes** and reinstall the app | | `is_archived` | The channel was archived | Unarchive it, or point `channel` at another channel | | HTTP `429`, `ratelimited` | More than about one message per second to one channel | Wait the seconds in `Retry-After` and post again; the example runbook does this once | | `no_text` | The message body had no `text` and no `blocks` | Check that the file Step 2 of the runbook wrote isn't empty | ## Security checklist - Keep the app to `chat:write`. Each extra scope widens what a leaked token can do: `chat:write.public` lets it post in every public channel, and the history scopes let it read every channel the bot is in. - Invite the bot only into the channels it needs; removing it from a channel takes that channel away. - Keep the token only in the Jetty workspace environment. Never put it in the runbook text, the task's `vars` or a URL. - The runbook tells the agent never to print the token; keep that line if you adapt it. - Messages appear under the app's name, visible to everyone in the channel. Don't have a runbook post secrets, customer data or raw logs there. - To cut off access immediately, call [auth.revoke](https://docs.slack.dev/reference/methods/auth.revoke) with the token, or remove the app in Slack: click your workspace name, then **Tools & settings > Manage apps > Installed Apps**, select the app, **App Details > Configuration > Remove App**. A run in progress fails at its next Slack call. --- Source: https://jetty.io/docs/integrations/custom # Custom Integrations Jetty has no fixed list of supported services. An integration is a credential you store on your workspace and a runbook that declares it, so anything with an API, a CLI or an MCP server can be connected the same way. This guide is the general recipe behind the [GitHub](/docs/integrations/github), [Google Drive](/docs/integrations/google-drive) and [Slack](/docs/integrations/slack) guides; use it for a service that has no guide of its own, including your own application. The worked example connects a runbook to an internal orders API with two variables, `APP_BASE_URL` and `APP_API_TOKEN`. Swap in your service's names and endpoints. **Prerequisites:** - A way to mint a credential for the service: an API token, a webhook URL or a connection string - A Jetty workspace where you can set environment variables and create tasks ## How it works Every integration has the same four parts: | Part | Where it lives | What it does | | --- | --- | --- | | The credential | The service | Says what the runbook may do there. Mint one for Jetty, with the smallest scope that does the job. | | The variable | Your workspace's environment | Holds the value under a name, such as `APP_API_TOKEN`. | | The declaration | The runbook's `secrets:` frontmatter | Names the variables this runbook needs. Only declared variables reach the run's agent. | | The instructions | The runbook body | Tell the agent which variable to use for which call, and to check it before doing any work. | For what happens between the workspace and the sandbox, see [Secrets and Environment Variables](/docs/ai-workloads/secrets-and-environment-variables). ## Set up the integration ### Step 1: Mint a credential for the runbook Create a credential that belongs to the runbook, not to a person. A few rules hold for almost every service: - **Smallest scope.** Read-only if the runbook only reads; one project, repository or channel if the service can scope that narrowly. - **Long-lived, not interactive.** A run has no browser and nobody to click through a login. Use an API token, a service account key, a webhook URL or a connection string. For an OAuth-only service, mint a refresh token once and store it with the client id and secret, as the [Google Drive](/docs/integrations/google-drive) guide does. - **Revocable on its own.** A dedicated token can be rotated or revoked without touching anything else. ### Step 2: Choose the variable names Use upper-case names with underscores, prefixed with the service: `APP_API_TOKEN`, `HUBSPOT_TOKEN`, `DATABASE_URL`. Store non-secret configuration (a base URL, an account id) as its own variable beside the token, so the runbook never hard-codes an environment. Note Names ending in `_TOKEN`, `_API_KEY`, `_API_TOKEN`, `_API_SECRET`, `_SECRET_KEY` or `_ACCESS_KEY` are recognised as secrets wherever Jetty records a step's configuration, and their values are replaced with `[REDACTED]`. Prefer one of those endings for anything sensitive. ### Step 3: Store the values in your workspace There are three ways to set a workspace variable. All three write to the same place. **Web app.** Open **Settings → Integrations** ([/settings/integrations](/settings/integrations)) and add each name and value. **Your coding agent.** With the Jetty plugin or MCP server installed, ask the agent to store it. Keep the value in a file so it doesn't land in the chat: Text ``` Read the token in ~/orders-token.txt and store it as APP_API_TOKEN in my workspace. Also set APP_BASE_URL to https://orders.example.com. ``` **API.** Send a `PATCH` to the workspace's environment endpoint. The body is wrapped in `environment_variables`, and the call merges these keys into the variables already there: Shell ``` curl -X PATCH "https://flows-api.jetty.io/api/v1/collections//environment" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"environment_variables": {"APP_BASE_URL": "https://orders.example.com", "APP_API_TOKEN": ""}}' ``` To remove a variable, send its name with `null`: Shell ``` curl -X PATCH "https://flows-api.jetty.io/api/v1/collections//environment" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"environment_variables": {"APP_API_TOKEN": null}}' ``` ### Step 4: Declare the secrets in the runbook Important A workspace variable is not visible to a runbook until the runbook declares it. The agent receives the variables listed in the `secrets:` block of the frontmatter, plus the model provider's own key. Without the declaration the sandbox has no `APP_API_TOKEN`. RUNBOOK.md frontmatter ``` --- name: orders-digest version: 1.0.0 description: Fetch yesterday's orders from the orders API and write a one-page digest secrets: APP_BASE_URL: { env: APP_BASE_URL } APP_API_TOKEN: { env: APP_API_TOKEN } --- ``` Each entry is a name and, under `env:`, the environment variable the run delivers the value in. The frontmatter never holds a value. A declared secret the workspace hasn't set does not stop the run: Jetty writes a line to the run log and the agent starts without it, which is why the runbook checks for it first (Step 5). If you work through the Jetty plugin or MCP server, its `check-secrets` tool compares these names with what your workspace has set and returns `{configured, missing, ready}`, names only. ### Step 5: Write the runbook around the credential Three habits make an integration dependable: 1. **Preflight.** Check that each variable is set before doing any work, and stop with a clear message if not. 2. **Prove the credential.** Make one cheap authenticated call (a `whoami`, a ping, `SELECT 1`) so a bad token fails in seconds. 3. **Keep the value out of the output.** Tell the agent never to print the token or put it in a URL. Jetty records what the agent prints and writes, exactly as produced. RUNBOOK.md body ```` # Orders Digest You fetch yesterday's orders from the orders API and write a one-page digest. The API's base URL is in the environment as `APP_BASE_URL` and its token as `APP_API_TOKEN`. Never print, echo, or write the token to any file or log, never put it in a URL, and never run `env` or `printenv` without filtering it out. ## Step 1 — Preflight ```bash mkdir -p /app/results [ -n "${APP_BASE_URL:-}" ] || echo "MISSING APP_BASE_URL" [ -n "${APP_API_TOKEN:-}" ] || echo "MISSING APP_API_TOKEN" ``` If either is missing, write `/app/results/summary.md` saying which, and stop. ## Step 2 — Check the credential ```bash curl -sS -o /app/results/whoami.json -w '%{http_code}\n' \ -H "Authorization: Bearer $APP_API_TOKEN" "$APP_BASE_URL/v1/whoami" ``` Anything other than `200` is a `FAIL`: record the status code (never the token) in `/app/results/summary.md` and stop. ## Step 3 — Fetch the orders ```bash curl -sS -G -H "Authorization: Bearer $APP_API_TOKEN" "$APP_BASE_URL/v1/orders" \ --data-urlencode "date={{date}}" > /app/results/orders.json ``` ## Step 4 — Summary Write `/app/results/summary.md`: the order count, the total, the three largest orders, and `PASS` or `FAIL` with the failing call and its status code. ```` ## Connecting through an MCP server If the service has a remote MCP server, the runbook can attach it and let the agent use its tools. Name the secret that authenticates the server with `secret:`: RUNBOOK.md frontmatter ``` --- name: issue-triage version: 1.0.0 description: Triage new issues through the tracker's MCP server agent: claude-code mcp_servers: tracker: url: https://mcp.tracker.example.com/mcp secret: TRACKER_TOKEN secrets: TRACKER_TOKEN: { env: TRACKER_TOKEN } --- ``` At run time Jetty reads `TRACKER_TOKEN` from the run's environment and writes it as the server's `Authorization: Bearer` header in the agent's MCP configuration. Because Jetty consumes the secret, it is **not** also placed in the agent's shell environment. If the runbook body needs the same token for a direct API call, opt in: YAML ``` secrets: TRACKER_TOKEN: { env: TRACKER_TOKEN, expose_to_agent: true } ``` A server definition can't carry a credential itself: an `Authorization` header, or any header or `env` key that looks like a token, is dropped with a warning, because the definition is stored with the task. Use `secret:`. MCP support varies by runtime; see [Runtime Configuration](/docs/ai-workloads/runtime-configuration). ## Overriding a credential for one run To point one run at a different account or environment, pass `secret_params` with the run request. They are merged over the workspace's variables for that run only and are not stored with the run: Shell ``` curl -X POST "https://flows-api.jetty.io/api/v1/run//orders-digest" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "init_params": {"vars": {"date": "2026-11-09"}}, "secret_params": {"APP_BASE_URL": "https://orders.staging.example.com", "APP_API_TOKEN": ""} }' ``` A [schedule](/docs/guides/scheduling-ai-workloads) can carry its own `secret_params` the same way; the API never returns them once set. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Preflight reports `MISSING APP_API_TOKEN` | The variable isn't in the runbook's `secrets:` block, or isn't set in the workspace | Add the declaration ([Step 4](#step-4)) and check the workspace ([Step 3](#step-3)). The run log names a declared secret the run did not set | | The variable is set and declared, but still empty in the shell | An MCP server or a code-check source names the same secret with `secret:`, so Jetty withholds it from the agent | Add `expose_to_agent: true` to the entry | | The service answers `401` or `403` | The token is wrong, expired, or lacks a scope | Test the token from your own machine with the same call, then update the workspace value; the next run picks it up | | The declaration was added but the run behaves as before | The task still holds the earlier runbook text | Save the runbook to the task again, then rerun | | The MCP server's tools never appear | The chosen runtime doesn't load MCP servers, or the token was missing | Pick a runtime with MCP support and check the run log for the server's token line | ## Security checklist - One credential per integration, minted for Jetty, with the smallest scope that works. - Keep the value only in the workspace environment. Never put it in the runbook text, the task's `vars` or a URL. - Anyone who can read the workspace's settings can read its variables, and any runbook in the workspace can declare them. Put credentials with different audiences in different workspaces. - Keep the “never print the token” line in the runbook. - Rotate by updating the workspace value; revoke at the service to cut off a run in progress. --- Source: https://jetty.io/docs/jetty-and-friends # Jetty and Friends You don't have to run your agent _on_ Jetty to get value from Jetty. You can keep your agent and use Jetty as the independent evaluation layer around it. ## Bring your friends and framework Here are some examples: - [Bring Your Friends and Framework](/docs/jetty-and-friends/bring-your-friends-and-framework): Keep your agent in its own framework and let Jetty grade every run: the pattern, with Eve and Flue worked examples. - [Client SDK](/docs/integrations/sdk): @jetty/sdk for TypeScript: JettyClient, runAndWait, etc. - [MCP Server](/docs/integrations/mcp-server): 27 tools for Claude Code, Cursor, VS Code, Windsurf, Zed, Codex, and WebMCP. --- Source: https://jetty.io/docs/jetty-and-friends/bring-your-friends-and-framework # Bring Your Friends and Framework You don't have to run your agent on Jetty to get value from it. Keep the agent wherever it already lives and use Jetty as the **independent grader, store, and A/B harness** around it: your framework runs the agent; Jetty runs a separate _grader_ [runbook](/docs/ai-workloads/runbooks), stores the run, and scores it against an independent rubric. Because the grader is a runbook the agent under test never sees, you can swap frameworks, models, or providers and keep the exact same measuring stick. The only Jetty-specific code is a few [SDK](/docs/integrations/sdk) calls. Two worked examples, end to end, and the general shape: [Eve + Jetty Mount the @jetty/eve extension into a Vercel eve agent: live grading, a grade-steered experiment bandit, and an eval reporter. Catch a regression before it ships.](/docs/jetty-and-friends/bring-your-friends-and-framework/eve) [Flue + Jetty The complete flue-jetty walkthrough: grade every Flue agent run with an independent rubric and catch a regression before it ships.](/docs/jetty-and-friends/bring-your-friends-and-framework/flue) Not what you're looking for? To wire Jetty into your own app directly, use the [Client SDK](/docs/integrations/sdk). --- Source: https://jetty.io/docs/jetty-and-friends/bring-your-friends-and-framework/eve # Eve + Jetty You shipped an [eve](https://vercel.com/docs/eve) agent, built on Vercel's filesystem-first framework where _an agent is a directory_. You tweaked a prompt. Is it better or worse? You can't tell from one reply. That's the _run, check, fix, rerun_ loop, and Jetty is the **check**. eve owns the agent loop. Jetty grades each output and keeps it: every run becomes a [record](/docs/ai-workloads/runs) you can score, label, and diff across versions, so a regression shows up before a customer finds it. The grader is a Jetty [runbook](/docs/ai-workloads/runbooks) the agent under test never sees, so a model can't grade its own output and rubber-stamp a regression. The agent-side integration ships as [`@jetty/eve`](https://github.com/jettyio/jetty-sdk/tree/main/packages/eve), an eve [extension](https://eve.dev/docs/extensions) you mount in one file — live ingest of every turn, a grade-steered experiment bandit, and a scoreboard tool. The example is [`examples/eve-jetty`](https://github.com/jettyio/jetty-sdk/tree/main/examples/eve-jetty) in the SDK repo. It A/B-tests a help-desk **triage agent** in two configs (warm and terse), grading every reply and flagging the config that regressed: Text ``` TICKETS: 2 GRADER: rubric (independent) config pass avg $/run v1 (warm) 2/2 4.7 0.0093 ✅ v2 (terse) 0/2 2.7 0.0032 ❌ regressed → v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm). ``` ## eve already has evals. Where Jetty fits eve ships a real eval system (`defineEval`, `eve eval`, an LLM-as-judge, a CI deploy gate). Those are your **unit and integration tests**: you author the assertions, they run your scripted sessions, they gate a deploy. Jetty sits beside them: an **independent grader** the builder didn't write, and a **durable store** where every run is a labelled record you can diff across versions, models, and many agents, long after the CI job ended. Keep `eve eval` for “did this commit break a rule”; add Jetty for “which version is better, and is it improving over time.” ## Install Jetty plugs into eve two ways: **`@jetty/eve`**, an eve [extension](https://eve.dev/docs/extensions) the agent mounts (live grading, an experiment bandit, an eval reporter), and the plain [SDK](/docs/integrations/sdk) for harness-side orchestration: Shell ``` npm install @jetty/sdk @jetty/eve ``` Requires `@jetty/sdk` 0.2.0+ (for `gradeWithJetty`) and, for the `@jetty/eve` extension, `eve` 1.0+ on Node 24+. `@jetty/eve` 0.4.0 and later build against eve 1.0 and no longer mount on eve 0.x; stay on `@jetty/eve` 0.3.x if your agent is still on eve 0.25–0.26. ## Mount the extension An eve extension is an agent-shaped package mounted under `agent/extensions/`; its contributions compose into the agent under the mount's namespace. The example's entire agent-side integration is one mount file ([`agent/extensions/jetty.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/agent/extensions/jetty.ts)): TypeScript ``` // agent/extensions/jetty.ts — the file name is the namespace import jetty from "@jetty/eve"; export default jetty({ collection: process.env.JETTY_COLLECTION ?? "", // empty → extension no-ops task: "triage-live", judgeMode: "simple_judge", // grade inline with a native Jetty judge arms: { warm: "…", terse: "…" }, // your reply styles; the bandit is included contract: "Respond with ONLY the triage JSON object { … }", }); ``` What the mount contributes, under the `jetty` namespace: | Contribution | What it does | | --- | --- | | `jetty__ingest` (hook) | Every finished turn lands in Jetty as a durable, labelled run — ungraded for an out-of-band grader, or judged inline by a native `simple_judge` task. | | `jetty__arm` (hook + dynamic instructions) | An episodic Thompson-sampling bandit over your configured reply-style arms, rewarded by the live pass-rates read back from Jetty labels. Grades steer traffic. The hook picks one arm when a turn starts; the instructions reapply it before every model call in that turn. | | `jetty__experiment` (tool) | The model can read its own scoreboard: per-arm judged runs, pass rates, and the current leader. Ask the agent “how is your experiment going?” | Config is validated (zod) and binds once at the mount site; with no `JETTY_COLLECTION` everything degrades gracefully, so the agent runs without Jetty credentials. Source: [`packages/eve`](https://github.com/jettyio/jetty-sdk/tree/main/packages/eve) in the SDK repo. ## The grading flow eve's agent is a directory you run with `npx eve dev` (or deploy to Vercel). Drive it over the typed `eve/client`, then hand the reply to a Jetty grading task and wait for the run. The grade comes back as a row you can label and compare: TypeScript ``` import { Client } from "eve/client"; import { JettyClient, gradeWithJetty } from "@jetty/sdk"; const eve = new Client({ baseUrl: process.env.EVE_URL ?? "http://127.0.0.1:2000" }); const jetty = new JettyClient(); // JETTY_API_TOKEN from env or ~/.config/jetty/token // 1. eve runs the agent (it owns the loop). const { delivery } = await eve.sessions.create({ message: prompt }); const turn = await delivery.result(); // 2. Jetty grades it server-side, with a grader that isn't the author — // upload, run the grader, read the grade, and label, in one call. const { grade, trajectoryId } = await gradeWithJetty(jetty, "acme", "triage-grader", { files: [{ filename: "case.json", data: turn.reply ?? "" }], labels: (g) => ({ "eval.grade": String(g.total) }), // labels can read the grade }); ``` Each grade is a Jetty run (the API calls runs trajectories): the inputs, outputs, grade, and cost, ready to replay. Compare the `eval.*` labels across configs to see which version slipped. > **Native reporter.** For eve's own evals, the `Jetty()` eval reporter drops into `evals.config.ts` exactly where the built-in `Braintrust(...)` reporter goes, so every `eve eval` result lands in Jetty automatically: `import { Jetty } from "@jetty/eve/reporter"` → `defineEvalConfig({ reporters: [Jetty()] })`. With no `JETTY_COLLECTION` set it no-ops, so the config is safe to commit. ## Configure | Variable | Required | Purpose | | --- | --- | --- | | `JETTY_API_TOKEN` | yes | Jetty API token (also read from `~/.config/jetty/token`). | | `JETTY_COLLECTION` | yes | Workspace that owns the grading task. | | `JETTY_GRADE_TASK` | yes | The grading runbook (e.g. `triage-grader`). | | `EVE_URL` | for the agent | Where the eve agent is reachable (`npx eve dev` serves `127.0.0.1:2000`). | | `AI_GATEWAY_API_KEY` / `VERCEL_OIDC_TOKEN` | for the agent | eve resolves models through Vercel AI Gateway, so the agent needs one of these. | | `OPENROUTER_API_KEY` | alt. to gateway | No AI Gateway? `agent/agent.ts` routes the agent through OpenRouter directly, an AI SDK “external” provider that bypasses the gateway (eve's `model` field accepts any AI SDK language model). | > **Credentials.** Put anything sensitive in `secretParams`, which the server keeps out of the stored run. Don't put secrets in `initParams`; that field is persisted. The SDK never logs your token. Tokens resolve from a constructor arg, then `JETTY_API_TOKEN`, then `~/.config/jetty/token`. ## What Jetty captures | eve | Jetty | | --- | --- | | Agent output (the draft) | The input the grading runbook scores | | Grade (1–5) | Label `eval.grade` on the run | | Pass / fail vs. the bar | Label `eval.pass` | | Per-run cost (estimated from model `usage.recorded` token usage) | Label `eval.cost_est_usd` | | Which agent config / version | Label `eval.config` | | The whole graded run | A run: inputs, outputs, steps, replayable | > eve reports token usage but no dollar cost, so the example _estimates_ `$/run` from tokens and a small per-model price table (`src/cost.ts`). Tune it to your real rates. ## Full walkthrough: catch a regression A step-by-step run of the example: first offline (no keys, ~10 seconds), then live. There are three levels of _do I need a key?_: 1. **Offline demo** (`npm run demo`): no keys, no eve, no network. 2. **Grading on Jetty**: covered by the **free trial** (12 runs, auto-activated). No API key. 3. **The live eve agent** runs via `npx eve dev` and resolves models through AI Gateway, so it uses your AI Gateway credential. ### 1\. Clone and build Run the example bundled in the repo. The `-w @jetty/sdk` flags refer to this monorepo's workspaces, so clone it and run everything from inside the checkout. (To use the SDK in your _own_ app instead, jump to [Use the SDK in your own project](#use-the-sdk-in-your-own-project).) Shell ``` git clone https://github.com/jettyio/jetty-sdk.git cd jetty-sdk npm install # installs the SDK, the example, and eve (one workspace install) npm run build -w @jetty/sdk # the example imports the built SDK cd examples/eve-jetty ``` ### 2\. Run the offline demo (no keys) Shell ``` npm run demo ``` You should see the verdict table immediately: Text ``` Acme Helpdesk — did my last change to the triage agent make it worse? (simulated; run `npm run ab-eval` against a live `npx eve dev` for the real thing) TICKETS: 5 GRADER: rubric (independent) config pass avg $/run ------------ ----- ---- ------- v1 (warm) 5/5 4.5 0.0051 ✅ v2 (terse) 1/5 3.5 0.0039 ❌ regressed → v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm). ``` It also writes `report.html` and opens it: a styled verdict and per-run breakdown, the same report the live run produces. This is a deterministic stand-in, no spend. If you only want to understand the example, you can stop here. ### 3\. Configure credentials (for the live run) Shell ``` cp .env.example .env ``` Edit `.env`: INI ``` AI_GATEWAY_API_KEY=... # or VERCEL_OIDC_TOKEN, for the eve agent JETTY_API_TOKEN=mlc_... # your Jetty token JETTY_COLLECTION=your-workspace # a workspace your token can write to JETTY_GRADE_TASK=triage-grader # leave as-is EVE_URL=http://127.0.0.1:2000 # where `npx eve dev` serves the agent ``` Load it into your shell (the scripts read `process.env`; they don't auto-load `.env`): Shell ``` set -a && . ./.env && set +a ``` ### 4\. Deploy the grader (one time) The harness calls a Jetty runbook that scores each draft. Deploy it into your workspace: Shell ``` npm run deploy-grader ``` This creates the `triage-grader` task from [`grader/RUNBOOK.md`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/grader/RUNBOOK.md) (and pushes a provider key into the workspace if you have one). Re-running it updates the task. ### 5\. Serve the agent and run the live A/B Shell ``` # terminal 1: serve the eve agent (needs Node 24+) npx eve dev # terminal 2: A/B-eval it (start small: each ticket is a real server-side grade) EVAL_TICKETS=2 npm run ab-eval ``` You'll see a line per run, then the verdict table: Text ``` v1 (warm) · reset: 4.7 PASS v1 (warm) · double-charge: 4.7 PASS v2 (terse) · reset: 2.7 fail v2 (terse) · double-charge: 2.7 fail TICKETS: 2 GRADER: rubric (independent) config pass avg $/run v1 (warm) 2/2 4.7 0.0093 ✅ v2 (terse) 0/2 2.7 0.0032 ❌ regressed → v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm). ``` …then it writes `report.html` and opens it: the verdict, a per-run breakdown, and links to each Jetty run. **The eval caught that the terse config regressed**, before it ever reached a customer. ### Run it on Jetty's free trial (no key) Jetty gives every workspace a free trial (12 runs, auto-activated, on Jetty's keys). When the workspace has no provider key, grading uses the trial automatically — no flag, no key-push: Shell ``` EVAL_TICKETS=2 npm run ab-eval ``` The trial covers **server-side Jetty runs** (the grader). The eve agent runs on your machine via `npx eve dev`, so the _full_ live run still needs an AI Gateway credential for the agent. But you can see Jetty's grading and runs on the trial, and `npm run demo` needs no keys at all. (Sonnet and most models are covered; Opus-class is excluded. After 12 runs, add your own key in Settings.) ### 6\. Inspect what got stored Every grade is a Jetty run, labelled with `eval.config`, `eval.grade`, `eval.pass`, and `eval.cost_est_usd`. View them in the Jetty UI (`https://flows.jetty.io//triage-grader`) or from code: TypeScript ``` import { JettyClient } from "@jetty/sdk"; const jetty = new JettyClient(); // reads JETTY_API_TOKEN const list = await jetty.listTrajectories(process.env.JETTY_COLLECTION!, "triage-grader", 5); for (const t of list.trajectories) { const full = await jetty.getTrajectory(process.env.JETTY_COLLECTION!, "triage-grader", t.trajectory_id); const labels = Object.fromEntries(full.labels.map((l) => [l.key, l.value])); console.log(t.trajectory_id, labels["eval.config"], labels["eval.grade"], labels["eval.pass"]); } ``` Because the runs are durable and labelled, you can compare configs across releases, long after the terminal session that produced them. ## How it works (the pieces) | File | Role | | --- | --- | | [`agent/instructions.md`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/agent/instructions.md) + [`agent/agent.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/agent/agent.ts) | The eve agent as a directory: the always-on system prompt and the runtime config. | | [`agent/extensions/jetty.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/agent/extensions/jetty.ts) | Mounts the [`@jetty/eve`](https://github.com/jettyio/jetty-sdk/tree/main/packages/eve) extension: live ingest, the arm bandit, and the `jetty__experiment` tool, configured in one place. | | [`src/tickets.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/src/tickets.ts) | The eval cases + the two configs (`v1` warm, `v2` terse). | | [`src/ab-eval.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/src/ab-eval.ts) | The live loop: for each config × ticket → drive eve via `eve/client` → grade + label → collect. | | [`src/cost.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/src/cost.ts) | Estimates per-run cost from eve token usage (eve has no dollar-cost field). | | [`src/eval.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/src/eval.ts) | `aggregate()` (per-config pass-rate/grade/cost) + `renderVerdict()` (the table). | | [`grader/RUNBOOK.md`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/grader/RUNBOOK.md) | The independent grader: a deterministic Python rubric. | The SDK does the orchestration: `runWithFiles`/`runAndWait` (with file upload), `getTrajectory`, `downloadFile`, `addLabel`, `createTask`. That's the part worth copying into your own eval. ## Use the SDK in your own project Everything above runs the example _inside this repo_. To use the SDK in a new, standalone project, you don't need this repo or any workspaces. Install the published package from npm: Shell ``` mkdir my-app && cd my-app npm init -y npm pkg set type=module # the SDK is ESM npm install @jetty/sdk @jetty/eve eve ``` The pattern isn't eve-specific: anywhere you can produce an agent output and call `jetty.runAndWait(...)` on it (eve, Flue, LangChain, a raw provider SDK, a hand-rolled loop), Jetty drops in as the eval layer. Copy the orchestration from [`src/ab-eval.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/src/ab-eval.ts) into your own code. ## Make it yours - **Add cases:** append to `TICKETS` in `src/tickets.ts`. - **Compare your own versions:** edit the two entries in `CONFIGS`, or change `EVE_MODEL` in `agent/agent.ts` to A/B models. - **Change the live arms:** the styles the bandit routes between are the `arms` config at the mount site (`agent/extensions/jetty.ts`) — add a fourth, reword one, or drop to two. - **Move the bar:** change `PASS_BAR` in `src/eval.ts` (batch A/B) or `passBar` at the mount site (live judge). - **Swap the grader:** the rubric in `grader/RUNBOOK.md` is plain Python. Replace it with an LLM-judge call for model-based grading, then `npm run deploy-grader`. - **Override a contribution:** turn the mount into a directory (`agent/extensions/jetty/extension.ts` + an override slot) to approval-gate or replace the extension's tool, hook, or resolver. ## Protect sensitive content Runs persist step inputs and outputs, so they're content-bearing. Put credentials in `secretParams` (kept out of the stored run), not `initParams`. If a draft can carry PII, redact it before grading, or grade a hash or summary instead. Treat run storage like any other logging surface. ## Troubleshooting - **The harness can't reach the agent.** Start it first with `npx eve dev` (Node 24+) and point `EVE_URL` at it. eve's HTTP channel fails closed for non-loopback traffic; for a deployed agent, add an authenticator in [`agent/channels/eve.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/agent/channels/eve.ts). - **The agent didn't return JSON.** The triage prompt asks for a bare JSON object; `extractTriage` tolerates fences and prose, but a chatty model can still wander. Tighten `agent/instructions.md`. - **`grader produced no /app/results files`**. The grader must run on a supported model, keep its `secrets` frontmatter, and write to `/app/results/`. All three are set in `grader/RUNBOOK.md`. - **The live run is slow.** Each grade spins up a sandbox (a few minutes for 2 tickets); that's expected, and the offline demo (`npm run demo`) is the fast path. > **Two complementary paths:** the batch A/B grades _through_ a Jetty task (Jetty executes the rubric, which creates the run) — Jetty as the independent _grader_. The `@jetty/eve` extension and the `Jetty()` eval reporter instead _push_ finished eve turns and eval results in via the run-ingestion endpoint — Jetty as the durable _scoreboard_. The example uses both. * * * ## See also - [The worked example](https://github.com/jettyio/jetty-sdk/tree/main/examples/eve-jetty) · [step-by-step tutorial](https://github.com/jettyio/jetty-sdk/blob/main/examples/eve-jetty/TUTORIAL.md) on GitHub - [`@jetty/sdk` on npm](https://www.npmjs.com/package/@jetty/sdk) · [SDK API reference](https://github.com/jettyio/jetty-sdk/tree/main/packages/sdk#readme) - [eve docs](https://vercel.com/docs/eve) · [Flue + Jetty](/docs/jetty-and-friends/bring-your-friends-and-framework/flue) · [Jetty and Friends](/docs/jetty-and-friends) --- Source: https://jetty.io/docs/jetty-and-friends/bring-your-friends-and-framework/flue # Flue + Jetty You shipped a [Flue](https://flueframework.com) agent. You tweaked a prompt. Is it better or worse? You can't tell from one reply. That's the _run, check, fix, rerun_ loop, and Jetty is the **check**. Flue owns the agent loop. Jetty grades each output and keeps it: every run becomes a [record](/docs/ai-workloads/runs) you can score, label, and diff across versions, so a regression shows up before a customer finds it. The grader is a Jetty [runbook](/docs/ai-workloads/runbooks) the agent under test never sees, so a model can't grade its own output and rubber-stamp a regression. The example is [`examples/flue-jetty`](https://github.com/jettyio/jetty-sdk/tree/main/examples/flue-jetty) in the SDK repo. It A/B-tests a help-desk **triage agent** in two configs (warm and terse), grading every reply and flagging the config that regressed: Text ``` TICKETS: 2 GRADER: rubric (independent) config pass avg $/run v1 (warm) 2/2 4.7 0.0093 ✅ v2 (terse) 0/2 2.7 0.0032 ❌ regressed → v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm). ``` ## Install There's no `flue add tooling jetty` package. Jetty plugs in through its [SDK](/docs/integrations/sdk): Shell ``` npm install @jetty/sdk ``` Requires `@jetty/sdk` 0.2.0+ (for `gradeWithJetty`). ## The grading workflow In a Flue workflow, draft with the agent, then hand the draft to a Jetty grading task and wait for the run. The grade comes back as a row you can label and compare: TypeScript ``` import { defineWorkflow } from "@flue/runtime"; import * as v from "valibot"; import { JettyClient, gradeWithJetty } from "@jetty/sdk"; import { triageAgent } from "../agent.js"; const jetty = new JettyClient(); // JETTY_API_TOKEN from env or ~/.config/jetty/token export default defineWorkflow({ agent: triageAgent, input: v.object({ ticket: v.any() }), async run({ harness, input }) { // 1. Flue runs the agent (it owns the loop). const session = await harness.session(); const draft = await session.prompt(JSON.stringify(input.ticket)); // 2. Jetty grades it server-side, with a grader that isn't the author — // upload, run the grader, read the grade, and label, in one call. const { grade, trajectoryId } = await gradeWithJetty(jetty, "acme", "triage-grader", { files: [{ filename: "case.json", data: draft.text }], labels: (g) => ({ "eval.grade": String(g.total) }), // labels can read the grade }); return { grade, gradeTrajectoryId: trajectoryId }; }, }); ``` Each grade is a Jetty run (the API calls runs trajectories): the inputs, outputs, score, and cost, ready to replay. Compare the `eval.*` labels across configs to see which version slipped. ## Configure | Variable | Required | Purpose | | --- | --- | --- | | `JETTY_API_TOKEN` | yes | Jetty API token (also read from `~/.config/jetty/token`). | | `JETTY_COLLECTION` | yes | Workspace that owns the grading task. | | `JETTY_GRADE_TASK` | yes | The grading runbook (e.g. `triage-grader`). | | `ANTHROPIC_API_KEY` | for the agent | The Flue agent runs on your machine, so it needs a model key. | > **Credentials.** Put anything sensitive in `secretParams`, which the server keeps out of the stored run. Don't put secrets in `initParams`; that field is persisted. The SDK never logs your token. Tokens resolve from a constructor arg, then `JETTY_API_TOKEN`, then `~/.config/jetty/token`. ## What Jetty captures | Flue | Jetty | | --- | --- | | Agent output (the draft) | The input the grading runbook scores | | Grade (1–5) | Label `eval.grade` on the run | | Pass / fail vs. the bar | Label `eval.pass` | | Per-run cost (`response.usage`) | Label `eval.cost_usd` | | Which agent config / version | Label `eval.config` | | The whole graded run | A run: inputs, outputs, steps, replayable | ## Full walkthrough: catch a regression A step-by-step run of the example: first offline (no keys, ~10 seconds), then live. There are three levels of _do I need a key?_: 1. **Offline demo** (`npm run demo`): no keys at all. 2. **Grading on Jetty**: covered by the **free trial** (12 runs, auto-activated). No API key. 3. **The live Flue agent** runs on _your_ machine via Flue, so it uses your Anthropic key. ### 1\. Clone and build Run the example bundled in the repo. The `-w @jetty/sdk` flags refer to this monorepo's workspaces, so clone it and run everything from inside the checkout. (To use the SDK in your _own_ app instead, jump to [Use the SDK in your own project](#use-the-sdk-in-your-own-project).) Shell ``` git clone https://github.com/jettyio/jetty-sdk.git cd jetty-sdk npm install # installs the SDK, the example, and Flue (one workspace install) npm run build -w @jetty/sdk # the example imports the built SDK cd examples/flue-jetty ``` ### 2\. Run the offline demo (no keys) Shell ``` npm run demo ``` You should see the verdict table immediately: Text ``` Acme Helpdesk — did my last change to the triage agent make it worse? (simulated; run `npm run eval` for the real thing) TICKETS: 5 GRADER: rubric (independent) config pass avg $/run ------------ ----- ---- ------- v1 (warm) 5/5 4.5 0.0051 ✅ v2 (terse) 1/5 3.5 0.0039 ❌ regressed → v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm). ``` It also writes `report.html` and opens it: a styled verdict and per-run breakdown, the same report the live run produces. This is a deterministic stand-in, no spend. If you only want to understand the example, you can stop here. ### 3\. Configure credentials (for the live run) Shell ``` cp .env.example .env ``` Edit `.env`: INI ``` ANTHROPIC_API_KEY=sk-ant-... # your Anthropic key JETTY_API_TOKEN=mlc_... # your Jetty token JETTY_COLLECTION=your-workspace # a workspace your token can write to JETTY_GRADE_TASK=triage-grader # leave as-is ``` Load it into your shell (the scripts read `process.env`; they don't auto-load `.env`): Shell ``` set -a && . ./.env && set +a ``` ### 4\. Deploy the grader (one time) The workflow calls a Jetty runbook that scores each draft. Deploy it into your workspace: Shell ``` npm run deploy-grader ``` Text ``` [env] pushed: ANTHROPIC_API_KEY [task] created your-workspace/triage-grader ✓ deployed: your-workspace/triage-grader ``` This pushes your `ANTHROPIC_API_KEY` into the workspace (so the grader's sandbox can run) and creates the `triage-grader` task from [`grader/RUNBOOK.md`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/grader/RUNBOOK.md). Re-running it updates the task. ### 5\. Run the live A/B Shell ``` npx flue run eval --target node --input '{"tickets":2}' ``` Each ticket is a real server-side grade (a sandbox run), so start with `tickets:2` (~a few minutes) before bumping to the full 5. You'll see a line per run, then the verdict table: Text ``` v1 (warm) · reset: 4.7 PASS v1 (warm) · double-charge: 4.7 PASS v2 (terse) · reset: 2.7 fail v2 (terse) · double-charge: 2.7 fail TICKETS: 2 GRADER: rubric (independent) config pass avg $/run v1 (warm) 2/2 4.7 0.0093 ✅ v2 (terse) 0/2 2.7 0.0032 ❌ regressed → v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm). ``` …then it writes `report.html` and opens it: the verdict, a per-run breakdown, and links to each Jetty run. **The eval caught that the terse config regressed**, before it ever reached a customer. ### Run it on Jetty's free trial (no key) Jetty gives every workspace a free trial (12 runs, auto-activated, on Jetty's keys). When the workspace has no provider key, grading uses the trial automatically — no flag, no key-push: Shell ``` # deploy without pushing a key (the trial covers the grader) unset ANTHROPIC_API_KEY npm run deploy-grader # grade on the trial — applied automatically when the workspace has no key npx flue run eval --target node --input '{"tickets":2}' ``` The trial covers **server-side Jetty runs** (the grader). The Flue agent runs on your machine, so the _full_ live run still needs your Anthropic key for the agent. But you can see Jetty's grading and runs on the trial with zero keys, and `npm run demo` needs no keys at all. (Sonnet and most models are covered; Opus-class is excluded. After 12 runs, add your own key in Settings.) ### 6\. Inspect what got stored Every grade is a Jetty run, labelled with `eval.config`, `eval.grade`, `eval.pass`, and `eval.cost_usd`. View them in the Jetty UI (`https://flows.jetty.io//triage-grader`) or from code: TypeScript ``` import { JettyClient } from "@jetty/sdk"; const jetty = new JettyClient(); // reads JETTY_API_TOKEN const list = await jetty.listTrajectories(process.env.JETTY_COLLECTION!, "triage-grader", 5); for (const t of list.trajectories) { const full = await jetty.getTrajectory(process.env.JETTY_COLLECTION!, "triage-grader", t.trajectory_id); const labels = Object.fromEntries(full.labels.map((l) => [l.key, l.value])); console.log(t.trajectory_id, labels["eval.config"], labels["eval.grade"], labels["eval.pass"]); } ``` Because the runs are durable and labelled, you can compare configs across releases, long after the terminal session that produced them. ## How it works (the pieces) | File | Role | | --- | --- | | [`src/tickets.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/src/tickets.ts) | The eval cases + the two configs (`v1` warm, `v2` terse). | | [`src/agent.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/src/agent.ts) | The Flue triage agent; the config's style is injected per prompt. | | [`src/workflows/eval.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/src/workflows/eval.ts) | The live loop: for each config × ticket → Flue draft → grade + label → collect. | | [`src/eval.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/src/eval.ts) | `aggregate()` (per-config pass-rate/score/cost) + `renderVerdict()` (the table). | | [`grader/RUNBOOK.md`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/grader/RUNBOOK.md) | The independent grader: a deterministic Python rubric. | | [`src/deploy-grader.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/src/deploy-grader.ts) | Deploys the grader via the SDK (`createTask` + `setEnvironmentVars`). | The SDK does the orchestration: `runWithFiles`/`runAndWait` (with file upload), `getTrajectory`, `downloadFile`, `addLabel`, `createTask`. That's the part worth copying into your own eval. ## Use the SDK in your own project Everything above runs the example _inside this repo_. To use the SDK in a new, standalone project, you don't need this repo or any workspaces. Install the published package from npm: Shell ``` mkdir my-app && cd my-app npm init -y npm pkg set type=module # the SDK is ESM npm install @jetty/sdk ``` JavaScript ``` // index.js import { JettyClient } from "@jetty/sdk"; const jetty = new JettyClient(); // reads JETTY_API_TOKEN (or ~/.config/jetty/token) console.log((await jetty.listCollections()).map((c) => c.name)); ``` There is **no `-w @jetty/sdk`** here. That flag only applies inside the monorepo. From there, copy the orchestration pattern from [`src/workflows/eval.ts`](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/src/workflows/eval.ts) into your own code. The pattern isn't Flue-specific: anywhere you can produce an agent output and call `jetty.runAndWait(...)` on it (LangChain, a raw provider SDK, a hand-rolled loop), Jetty drops in as the eval layer. ## Make it yours - **Add cases:** append to `TICKETS` in `src/tickets.ts`. - **Compare your own versions:** edit the two entries in `CONFIGS`, or change `FLUE_MODEL` to A/B models. - **Move the bar:** change `PASS_BAR` in `src/eval.ts`. - **Swap the grader:** the rubric in `grader/RUNBOOK.md` is plain Python. Replace it with an LLM-judge call for model-based grading, then `npm run deploy-grader`. ## Protect sensitive content Runs persist step inputs and outputs, so they're content-bearing. Put credentials in `secretParams` (kept out of the stored run), not `initParams`. If a draft can carry PII, redact it before grading, or grade a hash or summary instead. Treat run storage like any other logging surface. ## Troubleshooting - **`No workspaces found: --workspace=@jetty/sdk`** — you're not inside the jetty-sdk checkout. The `-w` flag is monorepo-only: clone the repo and run from its root (Step 1), or use the SDK in your own project (above). - **`grader produced no /app/results files`** — the grader must (a) run on a model the claude-code runtime supports (use `claude-sonnet-5`, not haiku), (b) keep the `secrets: ANTHROPIC_API_KEY` block in its frontmatter so the key reaches the sandbox, and (c) write to `/app/results/`. All three are already set in `grader/RUNBOOK.md`. - **`No Jetty API token found`** — you didn't load `.env`; run `set -a && . ./.env && set +a`, or export `JETTY_API_TOKEN`. - **`Cannot use import statement outside a module`** — `flue run` needs `"type": "module"` (already set here) and `flue.config.ts` at the example root. - **The live run is slow** — each grade spins up a sandbox (a few minutes for 2 tickets). That's expected; the offline demo (`npm run demo`) is the fast path. > **Note on scope:** this example grades _through_ a Jetty task, which is what creates the run. If your framework already produces a finished trace, you can instead push it directly with the SDK's `ingestTrajectory` (no workflow execution) — the [eve integration](/docs/jetty-and-friends/bring-your-friends-and-framework/eve) shows that pattern. * * * ## See also - [The worked example](https://github.com/jettyio/jetty-sdk/tree/main/examples/flue-jetty) · [step-by-step tutorial](https://github.com/jettyio/jetty-sdk/blob/main/examples/flue-jetty/TUTORIAL.md) on GitHub - [`@jetty/sdk` on npm](https://www.npmjs.com/package/@jetty/sdk) · [SDK API reference](https://github.com/jettyio/jetty-sdk/tree/main/packages/sdk#readme) - [Client SDK docs](/docs/integrations/sdk) · [Runbooks](/docs/ai-workloads/runbooks) · [Runtime Configuration](/docs/ai-workloads/runtime-configuration) --- Source: https://jetty.io/docs/integrations/sdk # Client SDK: `@jetty/sdk` The SDK is for when Jetty is part of an application or a CI check rather than a chat session: a typed TypeScript client that runs workflows, reads back the resulting runs, downloads artifacts, and labels runs. It's built around the loop Jetty is good at, **run → check → fix → rerun**, so the ergonomic entry point waits for a result and hands you the outputs. ## Install Shell ``` npm install @jetty/sdk ``` Requires Node 18+. ## Authenticate Construct a client with no arguments and it resolves your token the same way every Jetty tool does: an explicit argument first, then `JETTY_API_TOKEN`, then `~/.config/jetty/token`. If none are present it throws `JettyConfigError` rather than failing later with a confusing 401. TypeScript ``` import { JettyClient } from "@jetty/sdk"; // Resolves token from arg → env → ~/.config/jetty/token const jetty = new JettyClient(); // …or pass it explicitly const jetty2 = new JettyClient({ token: process.env.JETTY_API_TOKEN }); ``` ## Run a workflow and get the result `runAndWait` submits a run and polls to completion, returning a fully-resolved `Trajectory` (the SDK's type for a finished run). Read step outputs straight off it: TypeScript ``` const run = await jetty.runAndWait("acme", "triage", { ticket }); const { category, priority, draft_reply } = run.steps.triage.outputs; ``` Need files in the run? Pass them through `runWithFiles` (or `runAndWait`'s `files` option) and they land in the sandbox for the agent to read. ## The client surface `JettyClient` mirrors the product's nouns. The most-used methods: - **Workspaces**: `listCollections`, `getCollection`, `getCollectionEnvironment`, `setEnvironmentVars` - **Tasks**: `listTasks`, `getTask`, `createTask`, `updateTask`, `deleteTask` - **Runs**: `runWorkflow` (async), `runWorkflowSync`, `runWithFiles`, and the ergonomic `runAndWait` - **Run inspection**: `listTrajectories`, `getTrajectory`, `getStats` - **Labels & files**: `addLabel`, `downloadFile(storageKey)` → `{ bytes }` - **Schedules**: full CRUD plus `runRoutineNow` / `listRoutineRuns` (the SDK keeps the API's name for a schedule, _routine_) - **Step templates & trial**: `listStepTemplates`, `getStepTemplate`, `getTrialStatus`, `activateTrial` ## A/B evals Because `runAndWait` returns structured outputs and `addLabel` tags the run, the SDK is the natural way to grade one agent against another and detect regressions. That's exactly what the [agent-integrations](/docs/jetty-and-friends) example does: it runs an agent in another framework, then calls a Jetty grader runbook through the SDK and aggregates the scores. ## Python A Python SDK is planned but not yet shipped. Until it lands, Python projects can call the [REST API](/docs/reference/api) directly, since it's a small surface and the chat-completions endpoint is OpenAI-compatible. * * * Source: [jettyio/jetty-sdk](https://github.com/jettyio/jetty-sdk) (`packages/sdk`). See it drive a real eval in [Jetty and Friends →](/docs/jetty-and-friends) --- Source: https://jetty.io/docs/integrations/mcp-server # MCP server The Jetty MCP server is the npm package `jetty-mcp-server`. This page describes version 1.1.0. The server implements the Model Context Protocol (MCP). It gives an agent 27 tools that create, run, schedule, and inspect Jetty workloads. The server is a local process. Your MCP client starts it with `npx` and speaks to it over standard input and output (stdio). The server calls `flows-api.jetty.io` with your API token. Jetty does not host an MCP endpoint. The [Jetty plugin](/docs/integrations/bring-your-own-agent) for Claude Code, Codex, and OpenCode includes this server. The tool set is the same in every editor. ## Requirements - Node.js 18 or later. - A Jetty API token. Create one at [Settings → API keys](/settings/api-keys). A token starts with `mlc_`. ## Configuration The server reads two environment variables. | Variable | Required | Default | Function | | --- | --- | --- | --- | | `JETTY_API_TOKEN` | Yes | None | The API token the server sends with each request. | | `JETTY_API_URL` | No | `https://flows-api.jetty.io` | The base URL of the Jetty API. Set it only for a non-default backend. | If `JETTY_API_TOKEN` is not set, the server reads the token from `~/.config/jetty/token`. The `jetty login` command writes this file. The environment variable has priority over the file. ## Add the server to your editor The command is the same for every MCP client: `npx -y jetty-mcp-server`. Only the configuration file is different. ### Claude Code Run this command in the terminal: Shell ``` claude mcp add jetty -- npx -y jetty-mcp-server ``` Or add the server to `.mcp.json` in the project: JSON ``` { "mcpServers": { "jetty": { "command": "npx", "args": ["-y", "jetty-mcp-server"], "env": { "JETTY_API_TOKEN": "mlc_your_token_here" } } } } ``` ### Cursor Add the same object to `.cursor/mcp.json`. Windsurf reads the same structure from `~/.codeium/windsurf/mcp_config.json`. ### VS Code Add the server to `.vscode/mcp.json`. VS Code uses the key `servers`, not `mcpServers`: JSON ``` { "servers": { "jetty": { "command": "npx", "args": ["-y", "jetty-mcp-server"], "env": { "JETTY_API_TOKEN": "mlc_your_token_here" } } } } ``` ### Codex CLI Add the server to `~/.codex/config.toml`: TOML ``` [mcp_servers.jetty] command = "npx" args = ["-y", "jetty-mcp-server"] env = { JETTY_API_TOKEN = "mlc_your_token_here" } ``` ### Zed and other MCP clients Register a stdio server with the command `npx`, the arguments `-y jetty-mcp-server`, and the environment variable `JETTY_API_TOKEN`. Zed reads servers from the `context_servers` object in `settings.json`. Refer to the documentation of your client for the field names. ## Tools The server registers 27 tools. The names below are the names an agent calls. The API calls a run a _trajectory_ and a schedule a _routine_; the tool names keep the API terms. Each tool carries MCP annotations. 14 tools set `readOnlyHint: true`. They read data and change nothing. The other tools create or change data. Only `delete-routine` sets `destructiveHint: true`. | Group | Tool | Function | Read-only | | --- | --- | --- | --- | | Workspaces | `list-collections` | Lists all workspaces. | Yes | | Workspaces | `get-collection` | Gets one workspace and the names of its environment variables. | Yes | | Tasks | `list-tasks` | Lists the tasks in a workspace. | Yes | | Tasks | `get-task` | Gets one task and its workflow definition. | Yes | | Tasks | `create-task` | Creates a task from a workflow definition. | No | | Tasks | `update-task` | Changes the workflow or description of a task. | No | | Trial keys | `get-trial-status` | Gets the trial-key status of a workspace. | Yes | | Trial keys | `activate-trial` | Activates trial keys for a workspace. | No | | Runs | `run-workflow` | Starts a run and returns its workflow\_id immediately. | No | | Runs | `run-workflow-sync` | Starts a run and waits for the result (30 to 60 seconds is typical). | No | | Run inspection | `list-trajectories` | Lists the recent runs of a task. | Yes | | Run inspection | `get-trajectory` | Gets the full record of one run. | Yes | | Run inspection | `get-stats` | Gets the run statistics of a task. | Yes | | Run inspection | `add-label` | Adds a key=value label to a run. | No | | Step templates | `list-step-templates` | Lists the available workflow step templates. | Yes | | Step templates | `get-step-template` | Gets the details and input schema of one step template. | Yes | | Configuration | `check-secrets` | Compares the environment variables of a workspace with the variables a runbook needs. | Yes | | Configuration | `set-environment-vars` | Sets environment variables in a workspace. Existing keys are kept. A null value deletes a key. | No | | Schedules | `list-routines` | Lists the schedules of a workspace, or of one task. | Yes | | Schedules | `get-routine` | Gets one schedule, including its next fire time. | Yes | | Schedules | `create-routine` | Creates a schedule for an existing task. | No | | Schedules | `update-routine` | Changes the cadence, overrides, secrets, paused state, or webhook of a schedule. | No | | Schedules | `delete-routine` | Deletes a schedule. This cannot be undone. | No | | Schedules | `pause-routine` | Stops a schedule until it is resumed. | No | | Schedules | `resume-routine` | Resumes a paused schedule. | No | | Schedules | `run-routine-now` | Fires a schedule immediately and returns the workflow\_id of the run. | No | | Schedules | `list-routine-runs` | Lists the recent runs that a schedule started. | Yes | **Note:** `run-workflow-sync` holds the tool call open until the run completes. For a run that takes more than one minute, use `run-workflow`, then find the run with `list-trajectories` and read it with `get-trajectory`. ## Use the package as a library Version 1.1.0 adds importable entry points. All entry points are ES modules. The `npx jetty-mcp-server` command still starts the stdio server; editor configurations do not change. | Import | Runs in | Exports | | --- | --- | --- | | `jetty-mcp-server` | Node.js | Everything below, plus `JettyClient`: the API client that resolves the token from the environment or the token file. | | `jetty-mcp-server/api-client` | Browser and Node.js | `JettyApiClient`: the API client. You pass in the token, the API origin, and an optional URL builder. | | `jetty-mcp-server/tool-definitions` | Browser and Node.js | `JETTY_TOOLS`: each tool as data (name, description, zod input shape, annotations, handler). `JETTY_WEBMCP_TOOLS`: the subset that WebMCP exposes. | | `jetty-mcp-server/tools` | Node.js | `registerTools(server, client)`: registers the catalog on an `McpServer` from the MCP SDK. | | `jetty-mcp-server/webmcp` | Browser and Node.js | `findModelContext`, `registerJettyWebMcpTools`, `jettyWebMcpTools`, `toWebMcpTool`, and `toolInputJsonSchema`. See the next section. | ## WebMCP: the same tools in the browser [WebMCP](https://webmachinelearning.github.io/webmcp/) lets a web page give tools to an agent that runs in the browser. The page registers the tools on the model context of the document (`document.modelContext` or `navigator.modelContext`). On jetty.io, each signed-in app page registers the Jetty tools with the session of the signed-in user. There is nothing to install and no token to copy. A WebMCP-capable browser or extension can list tasks, start a run, or read a result on the page you have open. 2 tools are not available over WebMCP: `get-trial-status` and `activate-trial`. Trial keys are an account-level setting for the local agent that set up the workspace. The package never exposes these two tools to a browser agent, even when `include` names them. ### Register the tools on your own site Any site can register the catalog. Give `JettyApiClient` a function that returns the session token of the visitor, and a `buildUrl` function that maps each API path to a proxy on your own origin: TypeScript ``` import { JettyApiClient } from "jetty-mcp-server/api-client"; import { findModelContext, registerJettyWebMcpTools } from "jetty-mcp-server/webmcp"; const modelContext = findModelContext(); if (modelContext) { const client = new JettyApiClient({ token: () => session.getToken(), buildUrl: (path) => "/api/mise" + path.slice("/api/v1".length), }); const controller = new AbortController(); registerJettyWebMcpTools(modelContext, client, { signal: controller.signal, readOnly: false, include: ["list-tasks", "run-workflow", "get-trajectory"], }); } ``` **Note:** `flows-api.jetty.io` sends no CORS headers. A browser cannot call it directly. Route the calls through a proxy on your origin, or through a backend that adds the bearer token. jetty.io rewrites `/api/mise/*` to `/api/v1/*` on the API for this purpose. `registerJettyWebMcpTools` accepts these options: - `signal`: an `AbortSignal`. Abort it to unregister the tools. A signal that is already aborted registers nothing. - `include`: a list of tool names. Only these tools are registered. - `exclude`: a list of tool names. These tools are never registered. - `readOnly`: when `true`, only the tools with `readOnlyHint: true` are registered. - `namePrefix`: a string added to the front of each tool name, for example `jetty_`. The default is no prefix. The function returns the API shape it used (`registerTool` or `provideContext`) and the names of the tools it registered. Both shapes of the WebMCP API are supported. Each WebMCP tool keeps the name, description, and annotations of the MCP tool. Its `inputSchema` is JSON Schema, derived from the same zod shape. Its `execute` function validates the input, calls the API, and returns the same `{ content: [{ type: "text", text }] }` result as the MCP server. On failure it returns `isError: true` instead of throwing. An agent that knows the MCP server can use the page without new instructions. ## Transport The server runs on your machine as a stdio process. Your token goes from the process to the Jetty API and to no other server. There is no endpoint to deploy and no URL to configure. A guide that points an MCP client at an `https://…/mcp` URL for Jetty is out of date. Use `npx -y jetty-mcp-server`. * * * Source: [jettyio/jettyio-skills](https://github.com/jettyio/jettyio-skills) (`mcp-server/`). To call Jetty from an application, use the [Client SDK →](/docs/integrations/sdk) --- Source: https://jetty.io/guides/use-cases # Use Cases · How teams use Jetty AI Documentation menu # Use Cases Teams bring Jetty a job written in plain language, pick an agent and model, and let it run and be evaluated at scale. Here are some of their stories. - ## [Getting ahead of new model updates: automated testing across agents](/guides/getting-ahead-of-new-model-updates-with-primitive) ![](/_next/image?url=%2Fguides%2Fprimitive%2Fprimitive-logo.png&w=64&q=75)Primitive 1. On Jetty 2. Author 3. Run 4. Investigate 5. Improve Engineering time saved per run $225/deployment Evals per run 5 To build confidence in Jetty’s deployment checks. Faster plugin validation 41% 20 minutes across agents, in parallel. - ## [Evals as branding: from the first note to the final frame](/guides/from-the-first-note-to-the-final-frame-with-dial0) ![](/_next/image?url=%2Fguides%2Fdial0%2Fdial0-logo.png&w=64&q=75)Dial0 1. On Jetty 2. Author 3. Run 4. Investigate 5. Improve Verified Outputs: thumbnails, etc. 1k Modalities 5 Audio, MIDI, sheet music, images and text. Runbook improvements 37 Sandbox hours 11h Historical articles 100% sourced With verified citations and metadata. - ## [A small eval beats a big rewrite: improving skills in development](/guides/small-evals-beat-big-rewrites-with-nous-research) ![](/_next/image?url=%2Flogos%2Fproviders%2Fnous.png&w=64&q=75)Nous Research 1. On Jetty 2. Author 3. Run 4. Investigate 5. Improve Runs +15k Hundreds of parallel runs monitored and organized. Avg. evaluations per run 8 Scored by rubrics and judges. Run time configs 30–90 min --- Source: https://jetty.io/guides/getting-ahead-of-new-model-updates-with-primitive # Getting Ahead of New Model Updates | Primitive Case Study | Jetty AI Primitive is building the decision layer for teams shipping code with agents. Primitive’s prim client plugs into the coding agents its customers already use: Claude Code, Cursor, Codex and others. But each agent behaves slightly differently, and each one can be updated on a different schedule. Since engineering teams are using a diversity of agents and models at the same time, Primitive needs to reliably deploy updates to prim that will support each member of their team and meet them where they work. > “When I use Jetty, I have a new level of trust in the behavior of the non-deterministic systems outside of my control.” ## 01 · Situation Taylor Hodge is leading engineering, and with every customer bringing their own coding agent, he needed to know whether each new version of Primitive’s plugin would work as expected. For example, a recent Codex update changed how life-cycle hooks worked, but nothing surfaced to their users. This caused the overall user experience to suffer. **Call, return, improve:** the parent launches the same workload under each runtime configuration, the verdicts and their evidence come back to one report, and each improvement goes into the next sweep. ## 02 · Solution Primitive’s continuous integration tools already ran synthetic prompts against live providers and used Jetty as a judge on the outputs. Taylor wanted more than that: every agent, and every model, taken through Primitive’s whole workflow end to end, with a clear account of where things stop matching expectations. Primitive deployed a grader runbook that scores single eval cases, and its source lives in Primitive’s repository. The sweep is a runbook that runs runbooks. One parent run launches the cell runbook once per agent and model, then gathers every verdict into a single investigation report. Jetty provided a delivery workflow that was easy to scale to new runtime configurations: - **Same runbook, different inputs:** only the agent, the model and the prim version change between runs, so a difference points at the agent, not the test. - **Evals at two levels:** each child run scores the agent on Primitive’s journey checks; the parent’s own checks fail the investigation if any run goes unreported or any failure goes unexplained. - **Evidence, not opinions:** a run that ends without a report is relaunched once, and every diagnosis in the report has to quote the runs’ own output. **A step that runs runbooks:** the parent’s second step launches the same child runbook for every agent, and the verdicts come back to step 3. ## 03 · Outcome The runbooks deployed on Jetty were able to catch silent failures before they degraded customer performance. Better still, the team was able to proactively report and review complex workloads across a battery of model and agent configurations, and where necessary, deep dive into individual runs to improve the plugin integration. The investigation runbook opens with a single grid of every check against every agent, then gives each failure a diagnosis quoted from the runs. Because the sweep is itself a runbook, it reruns in parallel and on demand: a dozen runs across several agents with results in minutes. **How a sweep reads:** one parent launches the same child runbook for each agent, three runs each. Every square is one check in one run; outlined clusters are diagnosed once. Illustrative agents and results, not Primitive’s data. --- Source: https://jetty.io/guides/from-the-first-note-to-the-final-frame-with-dial0 # Evals as branding: from the first note to the final frame | Dial0 Case Study | Jetty AI Jonathan has been a visual artist for two decades and a member of the Directors Guild of America for 15 years. He’s building [Dial0](https://dial0.app), a platform that gives independent filmmakers the tools to grow the industry outside the studio system. Jetty is used in two of Dial0's core workloads for an upcoming film called Radical Vermont that's premiering for the state’s 250th anniversary on July 8, 2027. > “Jetty lets me hand off complex, multistep workload and trust they’ll run reliably and consistently, so I can focus on creating.” ## 01 · Situation Dial0 needs to analyze and generate all kinds of content. For Radical Vermont, that meant two things: sheet music so a live band can perform the film’s score at the premiere, and evaluation metrics to catch AI drift and keep every shot consistent. - **Sheet music for a live band:** [Code checks](/docs/ai-workloads/agent-evaluations/code-checks) verify the song files against their recorded hashes. Gemini listens to the recording to pick out the instruments, then music21 turns the MIDI into notes on the score, and the result is judged based on five criteria until it clears the quality bar. - **Proof that each shot is getting better:** Code checks confirm both renders match, then Gemini acts as a [judge](/docs/ai-workloads/agent-evaluations/judges), grading the new render against the last one from −5 to +5 on quality and checking seven continuity categories for drift. Independent checks combine those grades into a final verdict so the model can’t grade itself into a pass. Dial0 turned both jobs into runbooks with [Jetty’s coding agent plugin](/docs/integrations/bring-your-own-agent), so their own coding agent could write, run and refine their runbooks themselves. ## 02 · Solution Both workloads run as [Jetty runbooks](/docs/ai-workloads/runs) with two ways in: either as a tool called by an agent, or the app triggers the run automatically from inside the Dial0 interface. That’s the same bring-the-job, we-handle-the-infra model Jetty runs on generally: describe the job once, [point it at your own agent](/docs/integrations/bring-your-own-agent), and Jetty tracks the run underneath it. - **[Phonograph](#phonograph):** song to sheet music, with a master recording in and an editable draft score and instrument parts out. - **[Cinescope](#cinescope):** render progress review, grading every new render of a shot against the last one. ## 03 · Outcome A small independent team can now produce band-ready sheet music and keep every shot consistent. Both run as repeatable jobs on Jetty, started from the app and checked on later. Image evaluations are tracked, so models and providers can be compared, and time and budget are saved by avoiding repeated failures and speeding up improvements. ## What’s a runbook? A [runbook](/docs/ai-workloads/runbooks) is a plain-language instruction file for a job you want done the same way every time. Jetty hands it to a coding agent in a fresh sandbox, checks the work, and keeps every input, step and output as a run you can open later. The Cinescope and Phonograph tabs above each show one of Dial0’s runs, start to finish. 1. 1InputsThe files the job needs. 2. 2StepsAn agent follows the steps in an isolated sandbox. 3. 3OutputsThe deliverables, kept with the run. 4. 4EvaluationsCode checks and a judge decide if it passed. [See how Jetty runs and evaluates AI workloads](/docs/product-overview)[All use cases](/guides/use-cases) --- Source: https://jetty.io/guides/small-evals-beat-big-rewrites-with-nous-research # A Small Eval Beats a Big Rewrite — Nous Research | Jetty AI Nous Research has one of the best agent harnesses for performing deep research and other common business tasks. To do this effectively, they rely on bundling the best skills with their model and harness. Researchers have recently demonstrated that applying an evolutionary algorithm to skills can take small, cheap and lightweight models and have them outperform the most expensive tools to date. > “Jetty helped me run hundreds of long agent trajectories in parallel, cleanly and organized for analysis.” ## 01 · Situation For Morgane at Nous Research, using Jetty to optimize the skill was the shortest path to making significant improvements by leveraging the stored runs and runtime configuration, paired with the evaluation framework that ships with Jetty AI Workloads. As the project scope increased, Morgane used Jetty to shift the job from her laptop to [managed agents running parallel](/docs/product-overview/investigating-agent-quality), with shared memory and failure modes that could be read and turned into evaluations for future runs. The key unlock was a research loop that kept every config and every run in context and ready for skill self-improvement. ## 02 · Solution Morgane was able to run and test her skill with different versions of the Hermes Agent. Jetty’s ability to support a range of models and allow her to bring her own MCP tools. That’s the same bring-the-job, we-handle-the-infra model Jetty runs on generally: pick your agent and model pair, and Jetty tracks the configs, outputs, and logs underneath it. Every run’s configuration was stored automatically. She launched runs of [30 to 90 minutes](/docs/ai-workloads/runtime-configuration), many at a time, and left them. Set the AI workload, close the laptop, walk the dog and pick it back up when it’s done. - **[Onboarding was quick](/docs/integrations/bring-your-own-agent):** she ran against her own agent and model. - **[Parallel runs and saved runtime configuration](/docs/ai-workloads/runs):** every result reproducible and traceable. - **[Answered questions along the way](/docs/product-overview/investigating-agent-quality):** find failure modes and pick the best changes to the eval suite. ## 03 · Outcome Each run’s failure modes told Morgane exactly what to adjust for the next iteration. She was able to quickly prove out different algorithms and ultimately settle on a variant of the GEPA optimization mechanism. The best part was the skill was ready the moment the model landed: no agent-sitting. Using Jetty meant that runs finished on their own, freeing Morgane to work on the next iteration instead of watching a terminal. --- Source: https://jetty.io/guides/research # Research · Papers Jetty has contributed to Documentation menu # Research Jetty works with research labs and standards groups on how AI systems get evaluated: how benchmarks are packaged, how evaluations are made reproducible, and how models are held accountable once they leave the lab. These are the papers and whitepapers we have contributed to, newest first. 1. May 2026 · arXiv preprint ## [Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations](https://arxiv.org/abs/2605.29786) A declarative metadata format that separates an evaluation task from its solution, so agents can rebuild a benchmark’s pipeline from the spec instead of replicating its source code. Authors Omar Benjelloun, Leonardo Martins Bianco, Isabelle Guyon, Thanh Gia Hieu Khuong, **Jonathan Lebensold**, Sebastian Lobentanzer, Luis Oala, Benedictus Kent Rachmat, Ihsan Ullah, Peyman Vahidi, Joaquin Vanschoren Affiliations Google DeepMind, ChaLearn, Université Paris-Saclay, **Jetty**, Mila – Quebec AI Institute, Helmholtz Munich, German Center for Diabetes Research, Technical University of Munich, Helmholtz AI, Brickroad, Eindhoven University of Technology 2. March 2026 · arXiv preprint ## [CUBE: A Standard for Unifying Agent Benchmarks](https://arxiv.org/abs/2603.15798) A protocol standard built on MCP and Gym so an agent benchmark can be wrapped once and used by any compliant platform for evaluation, RL training or data generation. Authors Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko, Aman Jaiswal, Kusha Sareen, Shailesh Nanisetty, Joan Cabezas, Manuel Del Verme, Omar G. Younis, Simone Baratta, Matteo Avalle, Imene Kerboua, Xing Han Lù, Elron Bandel, Michal Shmueli-Scheuer, Asaf Yehudai, Leshem Choshen, **Jonathan Lebensold**, Sean Hughes, Massimo Caccia, Alexandre Drouin, Siva Reddy, Tao Yu, Yu Su, Graham Neubig, Dawn Song Affiliations ServiceNow AI Research, Silverstream.ai, Dalhousie University, IBM Research, **Jetty**, McGill University, Mila, The University of Hong Kong, The Ohio State University, Carnegie Mellon University, UC Berkeley 3. November 2025 · Whitepaper ## [AI Beyond Metrics: Insights from SMASH 2025](https://drive.google.com/file/d/1fYmpiOESCeu9DMO_Iy7jQllDnUXHSIDB/view) Takeaways from the inaugural Symposium on Model Accountability, Sustainability and Healthcare, hosted at Mila, on judging AI by trust, governance and sustainability rather than benchmark scores alone. Authors Göktuğ BenderForeword by Jonathan Lebensold (Jetty), SMASH co-organizer. Partners Akinox, CDSI (McGill University), CHAI, CIFAR, Google Sustainability, The Google School for Leaders, **Jetty**, Mila, MLCommons, OBVIA --- Source: https://jetty.io/guides # Guides Updated October 7, 2026 Task-shaped walkthroughs for the work you actually do. Each guide starts from a job and shows how to run it on Jetty: what goes in the runbook, what the runs show, and what to change when one misses. To see the loop on another team's workload, start with the [use cases](/guides/use-cases). Connecting a runbook to GitHub, Google Drive or Slack has its own section: [Integrations](/docs/integrations). [Authoring Your First RunbookBrief an agent, read its draft, decide what done means, run it and change one thing. Then compose runbooks that launch other runbooks.](/docs/guides/authoring-your-first-runbook)[Build vs. Buy EvaluationWhat AI agent evaluation costs to build versus buy: the variables that matter, a framework for making the call, and how to test it against your own workload.](/docs/guides/ai-agent-evaluation-cost)[Building AI EvaluationsA way to know, before you ship, whether your agent is actually doing the job it was built to handle at the level of quality you expect.](/docs/guides/building-ai-evaluations)[Creating a BenchmarkPick one axis of comparison, write the runbook that defines the job, and run the task set against every candidate in parallel so the comparison is reproducible.](/docs/guides/creating-a-benchmark)[Defining Agent ChecklistsKeep your agent on task with a checklist: the conditions that define done, ticked and rechecked inside every run, then investigated across runs and kept honest in production.](/docs/guides/ai-agent-checklist)[Improving Agent RunbooksPull the latest runs with your coding agent, tighten the evals they expose, and let the optimize-runbook skill propose evidence-backed edits.](/docs/guides/improving-agent-runbooks)[Scheduling AI WorkloadsRun a runbook on a cron cadence so evals stay fresh as models and providers drift.](/docs/guides/scheduling-ai-workloads) --- Source: https://jetty.io/docs/guides/authoring-your-first-runbook # Authoring your first runbook The best way to write your first [runbook](/docs/ai-workloads/runbooks) is with an agent. You brief an agent, it drafts the file, and Jetty runs it. The part that stays yours is deciding what “done” means and reading what the first run did. This guide covers those four moves, then one pattern worth knowing early: a runbook that runs other runbooks. **Four moves to a first runbook:** you write the brief, an agent drafts the runbook, Jetty runs it and scores it, and the run tells you the one thing to change before the next run. ## 1\. Brief it like a person Pick a small job you already know how to judge. Say three things in plain words: the job, the input that will change from run to run, and what a finished result looks like. Text ``` Compare the pricing pages of Acme, Globex and Initech. Input: the three company names. I'll want to swap them later. Done: one markdown table, a row per plan, with the monthly price and the page each price came from. If a plan has no public price, say "contact sales". Don't guess. ``` Give the brief to **Pelly** on the Author Runbooks screen in the web app, or to your own coding agent with the `create-runbook` skill. Both are set up in [How to start with Jetty](/docs/get-started), and both hand back the same thing: one markdown file. ## 2\. Read the draft A first draft for that brief looks like this: RUNBOOK.md ```` --- version: "1.0.0" agent: claude-code model: claude-sonnet-5 --- # Pricing page comparison ## Objective Compare the public pricing of the companies in {{companies}} so a sales lead can read the result in two minutes. ## Parameters | Parameter | Template Variable | Default | Description | |-----------|-------------------|---------|-------------| | Companies | `{{companies}}` | *(required)* | Comma-separated company names | ## Steps 1. Find the pricing page of each company in {{companies}}. 2. Record every plan: its name, its monthly price, and the page it came from. 3. Write {{results_dir}}/comparison.md as one markdown table, a row per plan. A plan with no public price says "contact sales". ## Code Checks ### comparison-written — comparison.md exists and is not empty ```bash test -s {{results_dir}}/comparison.md ``` ## Checklist - [ ] Every company in {{companies}} has at least one row - [ ] Every price has its source URL beside it - [ ] No price was guessed ## Required output files - {{results_dir}}/comparison.md ```` Before you run it, read it for four things: - **The steps describe work.** “Record every plan and the page it came from” can be followed. - **Every input is named in the steps.** The agent reads the body of the file. A parameter that appears only in the table is ignored. - **The output is a file with a name.** A run is judged on what it leaves in `{{results_dir}}`. - **There is at least one check.** Without one, a run can only tell you that it finished. The frontmatter is a default you can override per run, so don't spend time on it yet. The full anatomy is on the [Runbooks](/docs/ai-workloads/runbooks) page. ## 3\. Decide what done means Start with two kinds of check and keep both short. A [code check](/docs/ai-workloads/agent-evaluations/code-checks) is a command Jetty runs after the agent has finished, so the agent can neither skip it nor misreport it: the file exists, the JSON parses, the totals add up. A [checklist](/docs/ai-workloads/agent-evaluations/checklists) is the list of conditions the agent confirms about its own work, fixing and re-checking before it reports. Write each line so that a stranger could answer it yes or no. Three lines you trust is better than a dozen open to interpretation. ## 4\. Run it, then change one thing Run it once and open the [run](/docs/ai-workloads/runs). Read the verdict, then open the output file yourself: a first run often passes its checks and is still wrong in a way you would spot in a few seconds. That gap is your next edit. Fix the step that produced the miss, or add the check that would have caught it, and run again. One change per run, so you know which change worked. When two or three runs in a row pass and you agree with them, the runbook is ready to be used. Our [guide on improving agent runbooks](/docs/guides/improving-agent-runbooks) picks up from there. ## When one runbook isn't enough: runbooks that run runbooks A deployed runbook is a named thing you can call: parameters go in, files and a verdict come out. That makes it a building block. Any step of another runbook can launch it, wait for it and read what came back, which means the agent in the parent run is orchestrating other agents, each in its own sandbox with its own checks. **A runbook that runs runbooks:** one step of the parent launches the same child runbook once per configuration. Each child is a full run with its own checks; the parent’s checks judge the investigation, starting with whether every child reported. Illustrative agents and results. Taylor Hodge at Primitive uses this to [test one plugin across every coding agent](/guides/getting-ahead-of-new-model-updates-with-primitive). A child runbook takes one agent and model through Primitive's whole workflow and scores it. A parent runbook, the investigation, launches that child once per agent and model, gathers every verdict and writes a single report that diagnoses each failure. A sweep is 13 runs and takes about 20 minutes, and because the sweep is itself a runbook it can be rerun on demand or put on a schedule. The launch is an ordinary step in the parent, calling the same API you would: Markdown ```` ## Step 2: Launch the child runbook For each line of {{results_dir}}/configurations.txt, start one run of the child and save the id the call returns: ```bash curl -fsS -X POST "https://flows-api.jetty.io/api/v1/run/{{workspace}}/{{child_task}}" \ -H "Authorization: Bearer $JETTY_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"init_params": {"agent": "", "model": "", "vars": {"companies": "{{companies}}"}}}' ``` Do not retry a launch that returned an id: a retry starts a second run. ```` The parent needs a Jetty API token to make that call: store it in the workspace and declare it under `secrets:` in the parent's frontmatter (see [Secrets and environment variables](/docs/ai-workloads/secrets-and-environment-variables)). The run endpoint and the one that reads a run back are in the [API reference](/docs/reference/api). Four habits keep a composition honest: - **Author the child first.** Get it passing on its own with the four moves above. A parent cannot fix a child that doesn't work alone. - **Same runbook, different inputs.** Change only the parameters or the runtime configuration between children, so a difference in the results points at the thing you varied. - **Check at two levels.** The child's checks judge the work. The parent's checks judge the investigation: every child reported, and every failure in the report is explained. - **Ask for evidence.** Have the parent quote the children's own output for every conclusion it draws, and relaunch once any child that ends without a report. The same shape covers a benchmark across models, one job fanned out over a hundred inputs, or a nightly investigation of yesterday's runs. [Creating a benchmark](/docs/guides/creating-a-benchmark) and [Scheduling AI workloads](/docs/guides/scheduling-ai-workloads) take it further. --- Source: https://jetty.io/docs/guides/ai-agent-evaluation-cost # How Much Does AI Agent Evaluation Cost? Build vs. Buy When everything is changing, it makes sense to want maximum flexibility over the tools that your business is running on. Now with coding agents, the trade-offs often come only after running in production. One thing is clear: the evaluation of your AI system is something that is core to your workloads, whereas the infrastructure to support this effort can often be acquired. ## Which variables matter most? Every piece of software has points of flexibility and constraints which ensure good design. For example, in our [use case with Nous Research](/guides/small-evals-beat-big-rewrites-with-nous-research), we fixed the workload objective (deep research), the sandbox environment and the agent, but adjusted the runbook and evals as a way of improving model performance. Conversely, when trying to pick the right model for the job in a document, it might make sense to fix everything but the agent and the model choice. Agent evaluation infrastructure should make it easy to change the following parts of an AI workload: - Workload definition (the runbook / workload definition) - Agent evaluations (e.g. code checks, checklists and judges) - Runtime configuration (model choice, agent harness, MCPs and skills) - Integrations (third-party services, RAG retrieval systems) - Sandbox environment (including software libraries) At the same time, there are a TON of choices to make and it can be daunting just to get started. Jetty provides you with a [great set of sensible defaults](/docs/ai-workloads/runtime-configuration) so you can focus on the part that nobody else can decide for you; the workload definition and the definition of quality. ## A framework for making the call Here are some questions to get you started when thinking about the buy vs build question for agent evaluation infrastructure. | Question | Leans build | Leans buy | | --- | --- | --- | | **How many runs per week, realistically?** | A handful, run manually is fine | Dozens to thousands, manual monitoring doesn’t scale | | **Who else needs to trust the results?** | Just you, on your machine | A team that needs a shared, reproducible record | | **How often does the model or agent change?** | Rarely, one fixed setup | Often, and portability matters | | **What’s the evaluation need?** | A pass/fail check or two | Multiple judges, rubrics, and code checks per run | | **Where’s the real bottleneck?** | Infrastructure you don’t have yet | Engineering time better spent on the research question | With coding agents today, building each component is doable for a small team, however most of the complexity in any software system lies at the intersection of each component working together in concert, at scale. Our experience onboarding existing teams is that getting this right can shift a team from their main quest for three to six months. Open source solutions are great, but they still require you to learn and maintain the stack yourself. The most important question is what’s the cost of building a system that includes sandboxing, cloud storage, AI gateways, third-party integrations, observability and the ability to track and change your evaluation criteria. ## The cost comparison Jetty's free tier (Scoop) covers twelve runs a month at no cost, enough to test the comparison for real before committing to anything. Squadron Platform, the paid tier, is a flat monthly fee for unlimited runs across a team of up to ten, with judges, rubrics, and your own AI gateway included. The engineering cost of building yourself can be significant: maintaining parallel execution, retry logic, config storage, an evaluation framework, and ensuring that new models and agents keep working over time. For a team running real volume, that math tends to resolve quickly in one direction. ## Try it against your own workload The fastest way to answer this for your own team is to skip the guessing and run the comparison the way Morgane did with [Hermes Agent](/guides/small-evals-beat-big-rewrites-with-nous-research): point a real agent skill at both approaches and see which one gets you to a trustworthy answer faster. [Sign up for Jetty](/pricing) and bring your own agent. --- Source: https://jetty.io/docs/guides/building-ai-evaluations # Building AI Evaluations Story time: one of the big sources of AI quality comes from reading social media posts when a new model drops and figuring out if the “vibes” are good. This is the state of AI evaluation at the frontier. We can do better; instead of waiting for social proof from social media, you can use agent transcripts to evaluations that form part of your AI workload to avoid the negative chatter in the first place. The key is selecting an agent evaluation tool that continually improves your production AI workload. Jetty is an AI evaluation tool that gives you a way to know, before you ship, whether your agent is actually doing the job it was built to handle at the level of quality you expect. You need a tool that’s going to tell you quickly where to focus your effort. To do that each run needs a score that you can quickly pull up so you can track defects and surface errors. From there, you launch an investigation–a deep dive into what really happened. Sometimes this means re-running the same thing under a different runtime configuration and measuring the change. Other times, this might entail changing your evaluations to shine a stronger light on where the agent is tripping up. This process is exactly what a good agent evaluation tool enables. Since running investigations is itself going to involve more runs, you need a suite of tools that can track these changes without you having to get involved in maintaining metadata. ## Marie Kondo Your Evals Do these AI evals spark joy? That’s the right question to be asking yourself when designing the agent evaluations. An agent can solve a task in a dozen valid ways, or fail in a dozen invalid ones, and a simple pass or fail check can't tell the difference. We talk about the swiss-cheese approach to testing in [tests are boolean, evals are a gradient](https://blog.jetty.io/p/tests-are-boolean-evals-are-a-gradient): you need a diversity of evaluations to gain sufficient intuition into where an AI workload is struggling. Evaluating an agent means asking a bigger question on every run: did it use the tools correctly, did it reason through the problem soundly, would a person actually accept what it produced? That judgment doesn't come from one test. It comes from running the agent many times, on many inputs, and looking at the pattern of what breaks and what doesn't. Just like the things we accumulate in life, to reach serenity, we need to let go. Some AI evaluations need to be discarded when they are superseded. For instance, if your AI workload is effectively extracting the text, the checks around this could be added noise when trying to parse dates correctly post-extraction. ## The many sources of joy from your AI evaluation The 5 Category checklist that matters. - **Task completion.** Did the agent finish the job with the outputs we expect? - **Tool and skill use.** If the agent has access to tools, like a skill, MCP server, is it using them the way you intended? If they are working around them, it will show up in Jetty’s agent logs. - **Failure modes, not just failures.** A run that fails tells you something happened, see [investigating agent quality](/docs/product-overview/investigating-agent-quality), which discusses how to create meaningful analysis. - **Judgment, not just scoring.** Use a mix of code checks, checklists and judges. Judges, particular, are a great way to direct further improvements. And a good eval setup will avoid overlap between evaluation tools. - **Scale.** One good run is a great start, but thousands of runs across a real range of inputs will really inform whether the agent is actually reliable. ### Imagine your ideal ~lifestyle~ AI Workload The pattern that works, whether you build it yourself or use a tool, is the same loop every time: [run the agent](/docs/ai-workloads/runs), see where it breaks, [investigate](/docs/product-overview/investigating-agent-quality), [improve and run](/docs/guides/improving-agent-runbooks) again. To do this quickly, launch [runs in parallel](/docs/ai-workloads/runs) that will form a benchmark of realistic inputs and outputs. This is your team’s insurance policy for when something looks off two weeks from now. ## Close Your Laptop and Start Your Journey This is the loop Jetty is built around. You [bring your own agent and model](/docs/integrations/bring-your-own-agent), and Jetty handles the infrastructure underneath: running long runs in parallel, storing every [runtime configuration](/docs/ai-workloads/runtime-configuration) automatically, and surfacing the failure modes from each run so you know what to fix next. That's the same bring-the-job, we-handle-the-infra model behind everything Jetty runs, which is why [agent evaluations](/docs/ai-workloads/agent-evaluations) sit at the center of the product instead of bolted on the side. Avoid being locked into one model or one harness. Teams using Jetty describe this way of working as being closer to the KonMari Method: regularly letting go of outdated evaluations and adopting improvements to the AI workload in light of experience. Start your own journey: close your laptop, and come back to something you can act on instead of a pile you have to sift through. ## Start Komono Evaluation If your agent has never been through a real eval loop, the fastest way to organize and categorize your evaluations is to run it with Jetty. [Sign up to get a free Jetty account](/sign-up) and run your first batch of evals today. --- Source: https://jetty.io/docs/guides/creating-a-benchmark # Creating a Benchmark with Jetty Here’s the uncomfortable truth about the benchmarks the ML community treats as ground truth: they’re often wrong. For example, a recent analysis [from a popular LLM benchmark](https://arxiv.org/abs/2406.04127), found that 6.5% of questions contain errors. For instance over half of the questions in one category had problems: wrong answers, ambiguous phrasing, missing context. Instead of relying on public benchmarks, Jetty gives you the tools to build a benchmark based on your AI workload. ## A worked example For the last year or so, [Simon Willison](https://simonwillison.net/tags/pelican-riding-a-bicycle/) has been asking every new model to generate an SVG of a pelican riding a bicycle and posting the results. It’s become the unofficial visual reasoning test for frontier models. So when people see the Jetty mascot, the assumption is that we picked the pelican as a wink at the benchmark. Simon’s test exercises the layout, anatomy of two distinct objects and challenges the model's ability to keep coherent state across hundreds of XML elements. It’s visual enough that you can tell at a glance whether the model got it. ![Animated sequence of seven SVG renders of a pelican riding a bicycle produced by GPT-6 Sol, one per hill-climb run. The poster-style scene, titled The Daily Drift, barely changes between frames: the wing moves onto the handlebar grip, the feet onto the pedals, a saddle appears, and in the final frame the bicycle frame turns blue.](/guides/creating-a-benchmark/pelican-gpt6-sol-iterations.gif) One runbook, with iterative improvement of the evaluations (using OpenAI GPT-6 Sol). View more examples on our [pelicans benchmark](https://pelicans.jetty.bot). ## Three steps to getting started with custom benchmarks ### Step 1: Decide what you're actually comparing A benchmark needs one clear axis of comparison. Here are some ones that keep coming up with our users: - **Model comparison:** same agent, same task, different model or provider, to find the best fit for cost and quality. - **Agent comparison:** same task and model, different agent runtime (claude-code, codex, antigravity, and the rest), to see which handles your workflow best. - **Runbook or skill comparison:** same model and agent, different instructions, to measure whether a prompt or skill change actually helped. It’s a good idea to pick one dimension before you build anything since a benchmark that changes two variables at once can't tell you which one moved the result. Since every run in Jetty includes the runtime configuration, alongside its inputs, outputs, and evaluation results, it becomes easy to do a deeper investigation with the same runs later. Start by looking at the pass rate, then where failures cluster, then cost and runtime alongside quality, since the tradeoff between a cheaper model and a higher pass rate is usually the actual decision. ### Step 2: Write the runbook that defines the job Collect the inputs your benchmark will run against, including the hard cases, not just the ones that always pass. If you're working on a deep research task, you may have ambiguous questions and ones with missing information. It’s best to use these as the files and parameters your runbook will reference or require as inputs to a run. A [runbook](/docs/ai-workloads/runbooks) is the spec for the benchmark task itself: one markdown file with YAML frontmatter for defaults, a setup section, the steps the agent follows, and an evaluations section for scoring. The frontmatter sets the starting point for agent, model, and sandbox, but none of it is locked in, which matters for the next step. The four parts, the frontmatter fields and the evaluation styles are covered in full on the [Runbooks](/docs/ai-workloads/runbooks) page. ### Step 3: Run & Investigate the results The [evaluations](/docs/ai-workloads/agent-evaluations) section is what turns a batch of runs into an actual benchmark instead of a pile of transcripts. Jetty gives you three tools, and a real benchmark usually needs more than one of them: - **Code checks** for anything with a deterministic answer: did the output file exist, is it valid JSON, does a number reconcile. These never flake, and a failed code check fails the run's verdict, so use them for anything that has a clear right answer. - **Checklists** for plain-language conditions the agent works through before finishing, like “the response names the specific policy it's citing.” - **Judges** for the subjective quality question a code check can't answer. An LLM judge is fast and cheap for a plain scoring question; an agent judge, a separate grader run with tool access, costs more but catches things a single model call misses. A benchmark that only uses one of these is usually missing something: code checks alone miss quality, judges alone miss the objective failures a script would catch instantly. Run the full task set against each candidate, in parallel, so the comparison happens under the same conditions at the same time rather than across however many days it takes to click through it manually. Schedule future runs on a cadence, or trigger them with a webhook whenever a prompt, model, or skill changes, so that drift shows up automatically instead of getting discovered by an angry user. The [same runbook](/docs/ai-workloads/runbooks) that defined the one-off comparison is the one that keeps running, unchanged, every time you need to check whether something got better or worse. ## Where this fits with everything else Building a benchmark this way uses the same discipline behind [evaluating runs at scale](/docs/product-overview/investigating-agent-quality) and [comparing models](/docs/ai-workloads/runtime-configuration): a fixed task set, evaluation built into the workload instead of bolted on after, and every configuration stored automatically so the comparison is reproducible instead of a one-time impression. The benchmark isn't a separate system sitting next to your agent work, it's the same runbooks and runs you're already using, pointed at a fixed task set on purpose. ## Build one Start with one runbook, one task set, and one axis of comparison. [Sign up for Jetty](/sign-up) and turn your next “which model is actually better” argument into a run you can point to. ## Frequently asked questions ### How do I choose what to compare in a custom benchmark? Choose one clear comparison axis before building the benchmark. You can compare models with the same agent and task, agents with the same task and model, or runbooks and skills with the same model and agent. Changing two variables at once makes it impossible to tell which change moved the result. ### Why should the task set stay fixed when comparing AI systems? Keep the task set fixed so each candidate faces the same workload. Include hard cases, not only tasks that usually pass, and run the full set against each candidate. A fixed set makes the comparison reproducible, while the stored inputs, outputs, evaluations, and runtime configuration let you investigate the same runs later. ### Can I compare models, agents, and runbooks with the same benchmark? Yes, but each comparison should change only the dimension you are testing. For a model comparison, keep the agent and task the same; for an agent comparison, keep the task and model the same; for a runbook or skill comparison, keep the model and agent the same. This separates model, runtime, and instruction effects. ### Should a benchmark use code checks, checklists, and judges together? Use more than one evaluation method when the workload needs both correctness and quality. Code checks handle deterministic conditions, checklists cover plain-language requirements, and judges assess subjective quality. Code checks alone might not give the reason for a failure, while judges alone can miss objective failures that a script would catch immediately. ### How should I interpret pass rate, cost, and runtime together? Start with pass rate and inspect where failures cluster, then compare cost and runtime alongside quality. A higher pass rate may come with a different cost or runtime, so the useful result is the tradeoff between those measures. Jetty stores the runtime configuration with each run to support that deeper investigation. ### How can I repeat a benchmark and catch quality drift over time? Keep the runbook unchanged and schedule future runs on a cadence, or trigger them with a webhook when a prompt, model, or skill changes. Repeating the same task set lets you see whether a change improved or worsened results, and helps surface drift before it is discovered by a user. --- Source: https://jetty.io/docs/guides/ai-agent-checklist # Defining Agent Checklists A big part of investigating the quality of an AI workload is understanding how its evaluations performed over a period of time. This guide covers one of the most powerful tools you can define to get quick gains in an AI workload: the agent checklist. ## What is an AI agent checklist? Checklists in Jetty describe conditions that determine completion of the AI workload. Each item is something the agent can verify for itself before it stops. For example: a CSV has one row per input image, or the summary names every flagged receipt. The agent ticks through the list, fixes what fails, rechecks, and only then writes a report with an overall pass or fail. The checklist is one of three instruments Jetty uses to judge a run. Code checks fail a run outright, checklist items keep the agent on track while it works, and judges score what passed after the run. ## Closing the loop before you ship “It works on my machine” is a classic trope in software engineering. We’ve spent two decades moving away from a laptop-centric testing model and these days agents are bringing us back to this mode of operation: running agent loops for hours and hoping that the laptop doesn’t die between 5 PM and 9 AM. A checklist is a key part of keeping the agent on task and focused. Without a clear definition of done, and acceptance criteria encoded as a list of requirements at the end of the workload, agents will often arrive at open-ended results and unpredictable outcomes. How do you take an open agent loop and close it around a well-defined outcome? Here are some steps to get started: 1. **Set an objective.** Not “does it respond”, but what a correct, complete answer looks like, and what an unacceptable one looks like. If you can't describe the difference, you can't evaluate it, and neither can anyone else on your team. 2. **Turn experience into guards against common failures.** Check for hallucinated tool calls, half-followed instructions, early exits, and endless loops. Every AI workload accumulates its own history of failures, and ready to become another guard in a future checklist. 3. **Run an investigation and measure quality.** Assemble representative tasks, including hard or adversarial ones, so you can see which model and runtime configurations perform best. The checklist is part of every agent run, so you can use the results as part of deciding whether the AI workload is performing correctly. ## Running an agent investigation Investigations in Jetty are framed around a simple question. For instance, “which model is cheaper for this task?” or “which tasks performed the best/worst and why?”. Investigations allow you to retrospectively analyze your AI workload performance. Key facts about investigations: - **They run in parallel over a batch.** A single [run](/docs/ai-workloads/runs) tells you what happened once. Dozens of runs across your eval suite tell you what happens generally. - **They judge quality, not just completion.** Some tasks need more powerful evaluations, such as LLM judges and rubrics. [Tests are boolean, evals are a gradient](https://blog.jetty.io/p/tests-are-boolean-evals-are-a-gradient) explains how to pick the right judgment for each part of the task. - **They can measure across models, agents, MCP servers, and [runtime configuration](/docs/ai-workloads/runtime-configuration).** If your whole setup only works with one specific model, you don't have a robust agent, you have a fragile one that happens to work today. Jetty is model-agnostic, so one eval suite covers every configuration, and every run is recorded so you can reconstruct how an outcome occurred instead of guessing. - **They help maintain eval standards over time.** When you change the task, the tools, or the target behavior, your eval suite needs to change with it, and you need to know which version of the suite produced which results. [Improving agent runbooks](/docs/guides/improving-agent-runbooks) covers versioning both together. When your investigations start to yield insights into how your AI workload is performing, you're ready to improve performance in production. ## Improving performance in production In production, AI workloads are typically triggered by an API call, an MCP server, or another agent. Four strategies keep agents honest once they are there: - **Run investigations after every meaningful change.** Whenever a new model version drops, or a new condition is required, use the evaluations to benchmark the change before it reaches production. - **[Schedule the workload](/docs/guides/scheduling-ai-workloads)** so evaluations run periodically and you are notified automatically. - **Watch for drift, not just outright failure.** An agent can stay technically functional while quietly getting worse at the parts of the job that matter most. We discuss this in our article on [valley-dodging](https://blog.jetty.io/p/valley-dodging-the-half-of-agent). - **Revisit your baseline periodically.** What counted as “good enough” six months ago might not anymore, especially if the models underneath your agent have changed. Update the bar on purpose instead of letting it drift on its own. ## Where Jetty fits Maintaining an agent checklist over several weeks is a challenge for any team with a growing number of AI workloads. That management challenge is exactly what Jetty is built to handle: the parts that don't scale are tracked and maintained automatically. You [bring your own agent and model](/docs/integrations/bring-your-own-agent) over MCP, run your eval suite against it, and Jetty tracks the rest underneath, the same bring-the-job, we-handle-the-infra approach behind everything Jetty runs. Human judgement is more important now than ever, and agent checklists are a simple and effective method of embedding expectations into long agent workloads. [Verifiable agents improve trust in complex systems](https://blog.jetty.io/p/verifiable-agents-engender-trust), and a checklist recorded on every run is what makes an outcome traceable. ## Run the checklist If you've been meaning to put your agent through a real eval process and haven't gotten past step one, that's normal, and it's exactly what Jetty is for. [Sign up to get a free Jetty account](/sign-up) and run your first checklist against real data today. --- Source: https://jetty.io/docs/guides/improving-agent-runbooks # Improving agent runbooks The first version of a runbook is a hypothesis. It says what you think the agent needs to be told, and what you think “done” looks like. The [runs](/docs/ai-workloads/runs) are the evidence. This guide is about closing that gap on purpose: reading the last few runs, tightening the evaluations they expose, and editing the runbook so the next run is measurably better. It is the hands-on companion to [Investigating Agent Quality](/docs/product-overview/investigating-agent-quality), and it assumes you already have a runbook that runs (see [Writing runbooks](/docs/ai-workloads/runbooks#writing-runbooks) if not). There are two places to do this work. The [Jetty agent skill](/docs/integrations/bring-your-own-agent) in your own coding agent is the fully supported path and the one this guide follows. Jetty Web also has an **Optimize** view that runs the same analysis against a single run and shows the proposed edits as a diff; it is currently in **beta**, so expect rough edges, and use the skill whenever you want to look across several runs or keep the file under your own editor's control. ## The loop 1. **Pull the runs.** Ask your coding agent to fetch the most recent runs for the runbook: their verdicts, their output files, and the agent's own log. 2. **Improve the evals first.** For every miss the runs show, ask which check would have caught it, and add that check to the evaluations the runbook already defines. 3. **Then improve the instructions.** Fix the step that produced the miss: a missing guardrail, an ambiguous instruction, a Common Fix the agent needed and didn't have. 4. **Re-run and compare.** Two or three runs, side by side with the ones you started from. Keep the change if the verdicts moved; revert it if they didn't. ## The fastest loop: the `optimize-runbook` skill With the [Jetty plugin](/docs/integrations/bring-your-own-agent) installed, the `optimize-runbook` skill runs the loop above for you. Point it at a runbook and a task and it fetches the last N runs, reads each one's status, duration, step outputs, errors, and `validation_report.json`, then looks for six patterns: - **Consistent failures**: a criterion or stage below threshold in more than half of the runs. - **Iteration waste**: steps that need two or three retries every time. - **Timeouts and bottlenecks**: runs that ran long, and which step ate the time. - **Divergent behaviour**: structurally different outputs from the same instructions. - **Missing guardrails**: errors the evaluation section never caught. - **Score plateaus**: rubric criteria that iterate without improving. For each pattern it proposes a specific edit, quoting the exact current text and the exact replacement, and cites the runs that motivated it. You choose to apply all, pick individually, or save the analysis as a report. Applied edits bump the runbook's `version` so the deployed copy and the file on disk stay distinguishable. Text ``` /optimize-runbook acme/triage --last 5 /optimize-runbook acme/triage --trajectories 2ce3ec8e,d02e276b ``` Three or more runs give it real patterns; one run gives it one data point. If you have only one, read it yourself first (the next section), then run the skill once more runs exist. ## Best practice: start from the runs, and improve the evals first The single most useful habit is to ask your coding agent to pull down the most recent runs from a runbook and use them as the basis for updating the evaluations that are already defined. Not to write new instructions from memory, and not to guess at what went wrong: to read what actually happened and make the checks catch it next time. Text ``` Pull the last 5 runs of acme/triage. For each one, read validation_report.json, summary.md, and the primary output. List every place a run fell short that no evaluation stage caught. For each, propose the check that would have caught it, added to the evaluation section that already exists in RUNBOOK.md. Don't rewrite the instructions yet. ``` Why evals first: an instruction change without a check is a hope. A check without an instruction change still fails loudly, which is exactly what you want while you work out the fix. And a check written against a real miss is specific in a way a check written on day one never is. The [evaluation styles](/docs/ai-workloads/agent-evaluations) all take this well: a new code check, a new checklist line, or a new judge criterion. Before you keep a new code check, run it against the outputs of a few recent _passing_ runs. If they fail it, the check is wrong, not the runs. ## What to read in a run Your agent can fetch all of this with the `jetty` skill or the [MCP tools](/docs/integrations/mcp-server) (`list-trajectories`, `get-trajectory`, and the file download endpoint). Ask for, in this order: - **The verdict.** `validation_report.json`: which stages passed, which failed, and whether `overall_passed` is honest about it. - **The deliverables.** The files named in `primary_outputs`. Open them. A stage can pass while the output is still wrong in a way you would spot in ten seconds. - **The summary.** `summary.md`: what the agent thought it did, and any limitations it admitted to. - **The agent's own log.** The tool-by-tool timeline shows where the minutes went and where the agent improvised. A run that took 25 minutes because 19 of them were spent retrying one tool call looks identical to a slow run until you read this. - **Duration and cost.** Compare against the step's `timeout_sec`. A run that finishes a minute under its timeout will not finish next week. ## Lessons from the field These are the changes that came out of real optimize passes. Each one started as a run that completed, looked fine at a glance, and wasn't. ### “Completed” is not “passed” A run can complete with a green status and zero output files, because the agent hit an error and stopped politely. Treat _completed with no results files_ as a failure, and make `overall_passed` in `validation_report.json` the signal you trust. Give it a `PARTIAL` category for degraded-but-usable runs (a fallback data source, a skipped optional input) so it never has to lie in either direction. ### Put the deterministic work in a script, and keep the agent's judgement small If a runbook queries an API, computes numbers, and renders a report, the queries, the arithmetic, and the rendering should be a script the agent runs, not steps the agent improvises. Runs become fast, cheap, and comparable, and the agent's effort goes where it is worth something: reading the numbers and writing the two paragraphs that need a person. A runbook that produces a report every day should produce the same report every day except for the sentences that need thought. ### Never make the agent retype a big file An early version of one report runbook inlined a 73 KB builder script in a code block and told the agent to write it to disk. The agent spent 19 of the run's 25 minutes failing to do that, then pasted it in four chunks. Moving the script out of the runbook and mounting it as a file (upload it once, reference it in the task's `file_paths`, copy it from `/app/assets` at run time) cut the same run to two minutes. The runbook shrank from 91 KB to 20 KB, which also matters because the instruction has a hard size ceiling after parameter substitution, and past it the agent fails without a useful error. Keep runbooks small; keep code in files. ### Make the checks structural and cheap The checks that catch the most are the boring ones: two counts that must add up to a third, no unfilled template slot left in the output, no `None` in prose, no secret-looking string in a report, a note that cites at least one number per bullet, a file over a minimum size. They cost nothing to run every time, they never disagree with themselves, and each one encodes something a person once had to notice by eye. ### Turn every real failure into a Common Fix When a run fails on something specific (a query dialect quirk, a reserved word, a certificate bundle missing off-sandbox, a rate limit), the fix goes into the runbook's Common Fixes table as a row: the exact error text, the exact fix. The next run applies it in one retry instead of three. The `optimize-runbook` skill prefers these additive edits for a reason: they cannot make anything else worse. ### Watch the clock Read the agent duration on every run, not just the failed ones. When a run brushes its timeout, fix the cause first (usually one slow tool call or a retry loop) and only then raise `timeout_sec`. Raising the timeout on its own turns a fast failure into a slow one. ### Keep the verdict honest Bound every retry loop (three rounds is the convention) and make the runbook report what it could not fix rather than ship it. Write rubric criteria to describe the target, not the current output, because a rubric that describes what the runbook already produces will rubber-stamp it. When you disagree with a judge, label the run with your own verdict and watch the agreement rate; when it drifts, fix the judge before trusting another score. ### Version, redeploy, and check that you did A deployed task carries its own copy of the runbook. It is easy to improve the file on disk and leave version 1.0.0 running on the schedule. Bump the version on every structural change, redeploy, and read the deployed instruction back before you trust the next run. ## Then let it run Once two or three consecutive runs pass with the tightened evals, put the runbook on a cadence with [Scheduling AI Workloads](/docs/guides/scheduling-ai-workloads) and keep reading the verdicts. Models and providers drift; a check that passed for a month and starts failing is the runbook telling you something changed underneath it. That is the moment to pull the last few runs and start this loop again. --- Source: https://jetty.io/docs/guides/scheduling-ai-workloads # Scheduling AI Workloads A **schedule** runs a task or runbook on a cron cadence. Instead of kicking off a run by hand, you give it a cadence and Jetty fires it for you — on the hour, every morning, once a week, whatever you set. ## Why schedule There are two everyday reasons to put a task on a schedule: - **Keep evals fresh.** Models drift. Providers change defaults, deprecate versions, and re-tune behind the same name. A runbook that passed at 84% last month can quietly regress without a single line of your code changing. A nightly schedule re-runs the eval so you catch the drift instead of discovering it in production. See [Agent Evaluations](/docs/ai-workloads/agent-evaluations). - **Recurring jobs.** A daily report, a weekly digest, a scheduled data pull — anything you'd otherwise remember to trigger yourself. ## Create a schedule Schedules hang off a task. In the app, open the task and go to its **Schedules** tab; you can also create and manage them through the Schedules API (see below). A schedule has: - **A cadence**: a cron-like schedule that says when it fires. - **Controls**: pause it, resume it, or run it now without waiting for the next tick. Pausing is the safe way to stop a schedule temporarily; the schedule is preserved, so resume picks up exactly where it left off. Run-now is handy for confirming a schedule works before you trust it to run unattended. ## What a firing does A schedule fires the _same workload_ as a manual run of that job: same steps, same config, same evals. Each firing produces its own [run](/docs/ai-workloads/runs), so your scheduled runs show up in the same history as your manual ones and carry the same recorded eval result. ## The Schedules API Everything the Schedules tab does is available over the API: full CRUD on schedules plus run-now and listing past schedule runs. Use it to provision schedules from infrastructure code, or to wire run-now into your own tooling. The methods and endpoints are in the [API reference](/docs/reference/api) (the [SDK](/docs/integrations/sdk) exposes them too, as `runRoutineNow` and `listRoutineRuns`). The API, SDK and MCP server keep the older name for a schedule, _routine_: the endpoints live under `/api/v1/routines` and the tool and method names follow suit. * * * A scheduled eval is only useful if it's a good eval. See [Agent Evaluations](/docs/ai-workloads/agent-evaluations) for how scores get recorded, and the [API reference](/docs/reference/api) for the schedule endpoints. --- Source: https://jetty.io/docs/overview # Jetty documentation Jetty is where you **find, share, and run reliable AI workflows**. You write a [runbook](/docs/ai-workloads/runbooks) — one markdown file that gives an agent its job, its bar for “done”, and its checks — and Jetty runs it in a managed sandbox, records every run in full, and helps you improve it until it works every time. These docs are written for **builders**: developers who live in Claude Code, Codex, or Cursor and want a runbook running in minutes instead of a framework to stand up. Pick a starting point. ## Get started From zero to a runbook you trust, in one sitting. [Get StartedWhat Jetty runs, and three ways in: start free, install locally, or call it from your app.](/docs/get-started)[Try a ProjectBrowse the Jetty Runbook Directory and try it yourself on your own account.](/runbooks)[Bring Your AgentInstall Jetty locally: the plugin, the jetty-setup skill, and your first run.](/docs/integrations/bring-your-own-agent)[Connect Your AppEvery on-ramp: the MCP server, the Client SDK, and the API.](/docs/get-started/connect-your-app) ## Product overview What Jetty does, in two motions: run the workload, then read the run. [Product OverviewThe two halves of Jetty: run AI workloads in managed sandboxes, and investigate how well the agent performed.](/docs/product-overview)[Running AI WorkloadsHow a runbook becomes a running workflow: the author → run → investigate → improve lifecycle, and what Jetty assembles around every run.](/docs/product-overview/running-ai-workloads)[Investigating Agent QualityHow to read a run and its verdict, add a check at the failure moment, compare runs honestly, and run the investigate-and-improve loop.](/docs/product-overview/investigating-agent-quality) ## AI workloads The job you hand to Jetty, and the pieces that carry it out. [AI WorkloadsWhat counts as an AI workload, and the two pieces every one is built from: a runbook and a runtime.](/docs/ai-workloads)[RunbooksThe hero artifact: one markdown file with the job, the bar for “done”, and the checks.](/docs/ai-workloads/runbooks)[Runtime ConfigurationThe agents Jetty runs inside its own sandboxes, which providers back each, and how to pick one.](/docs/ai-workloads/runtime-configuration)[antigravityGoogle's current terminal agent (agy) in Gemini-API-key mode: catalog slugs, effort tiers, MCP support.](/docs/ai-workloads/runtime-configuration/antigravity)[claude-codeThe default runtime: Anthropic's agent with MCP, full usage telemetry, and the loop Jetty's skills are tuned against.](/docs/ai-workloads/runtime-configuration/claude-code)[codexOpenAI's agent on a pinned release: Responses-API only, config.toml routing, the reasoning-effort knob.](/docs/ai-workloads/runtime-configuration/codex)[hermesNous Research's agent; the one runtime that reaches all six providers, including self-hosted vLLM.](/docs/ai-workloads/runtime-configuration/hermes)[opencodeThe open-source SST agent with permissions pre-allowed and Playwright + Firecrawl MCP ready on every run.](/docs/ai-workloads/runtime-configuration/opencode)[piA small provider-agnostic agent with no MCP client by design: CLI tools and skills instead.](/docs/ai-workloads/runtime-configuration/pi)[SandboxesThe isolated machine each run gets: what the default sandbox contains, its lifecycle and limits, where files go, and how to run on your own Docker image.](/docs/ai-workloads/sandboxes)[Secrets and Environment VariablesWhere a secret is stored, how it travels from your workspace into the sandbox, which process receives it, and what Jetty keeps out of the recorded run.](/docs/ai-workloads/secrets-and-environment-variables)[RunsEvery run is captured end to end: each step, its inputs, outputs, files, and status, plus labels to group and compare runs.](/docs/ai-workloads/runs)[Agent EvaluationsAgent checks, Jetty checks, checklists and judges live in the runbook and run in flight. How the report comes together, the verdict, strict evaluation, and a failed run compared with a failed evaluation.](/docs/ai-workloads/agent-evaluations)[Code ChecksThe two kinds of code check, in the order they run: agent checks, run and reported by the agent, then Jetty checks, run by Jetty after the agent finishes. How to write each, and the sources and secrets a check can use.](/docs/ai-workloads/agent-evaluations/code-checks)[ChecklistsPlain-language conditions the agent confirms before it reports done, with a bounded fix-and-recheck loop.](/docs/ai-workloads/agent-evaluations/checklists)[JudgesLLM judges, agent judges and rubrics: a model or agent scoring the output against a criterion, with evidence.](/docs/ai-workloads/agent-evaluations/judges)[REST APIThe Jetty REST API: run runbooks as tasks (deploy, run, monitor — the primary endpoint), the OpenAI-compatible chat-completions compatibility layer, schedules, webhooks, and GitHub-PR APIs.](/docs/reference/api)[Machine InstructionsThe runbook API surface for agents and programmatic callers: tasks, runs, init\_params, trajectories, files, MCP config, sweeps, snapshots, webhooks, schedules.](/docs/reference/api/machine-instructions)[ActionsThe pre-built actions that run alongside runbooks — AI models, control flow, data processing, evaluation — and the path expressions that wire them together.](/docs/reference/api/steps) ## Jetty and Friends Already have an agent in another framework? Keep it. Jetty becomes the independent evaluation, observability, and optimization layer around it, or you run one of the four built-in runtimes inside Jetty's own sandboxes. [Jetty and FriendsKeep your agent in any framework; use Jetty as the independent grader, store, and A/B harness. Flue and eve worked examples.](/docs/jetty-and-friends)[Bring Your Friends and FrameworkKeep your agent in its own framework and let Jetty grade every run: the pattern, with Eve and Flue worked examples.](/docs/jetty-and-friends/bring-your-friends-and-framework)[Eve + JettyMount the @jetty/eve extension into a Vercel eve agent: live grading, a grade-steered experiment bandit, and an eval reporter. Catch a regression before it ships.](/docs/jetty-and-friends/bring-your-friends-and-framework/eve)[Flue + JettyThe complete flue-jetty walkthrough: grade every Flue agent run with an independent rubric and catch a regression before it ships.](/docs/jetty-and-friends/bring-your-friends-and-framework/flue)[Client SDK@jetty/sdk for TypeScript: JettyClient, runAndWait, etc.](/docs/integrations/sdk)[MCP Server27 tools for Claude Code, Cursor, VS Code, Windsurf, Zed, Codex, and WebMCP.](/docs/integrations/mcp-server) ## Go deeper The task-shaped guides, when you want them. [GuidesEvery guide in one place: building evaluations and benchmarks, defining checklists, improving a runbook from its own runs, and scheduling workloads.](/guides)[Authoring Your First RunbookBrief an agent, read its draft, decide what done means, run it and change one thing. Then compose runbooks that launch other runbooks.](/docs/guides/authoring-your-first-runbook)[Build vs. Buy EvaluationWhat AI agent evaluation costs to build versus buy: the variables that matter, a framework for making the call, and how to test it against your own workload.](/docs/guides/ai-agent-evaluation-cost)[Building AI EvaluationsA way to know, before you ship, whether your agent is actually doing the job it was built to handle at the level of quality you expect.](/docs/guides/building-ai-evaluations)[Creating a BenchmarkPick one axis of comparison, write the runbook that defines the job, and run the task set against every candidate in parallel so the comparison is reproducible.](/docs/guides/creating-a-benchmark)[Defining Agent ChecklistsKeep your agent on task with a checklist: the conditions that define done, ticked and rechecked inside every run, then investigated across runs and kept honest in production.](/docs/guides/ai-agent-checklist)[Improving Agent RunbooksPull the latest runs with your coding agent, tighten the evals they expose, and let the optimize-runbook skill propose evidence-backed edits.](/docs/guides/improving-agent-runbooks)[Scheduling AI WorkloadsRun a runbook on a cron cadence so evals stay fresh as models and providers drift.](/docs/guides/scheduling-ai-workloads) ## Use cases Stories from teams running AI workloads on Jetty. [Use CasesStories from teams running AI workloads on Jetty: what they set up, how parallel runs and evals changed the work, and what came out.](/guides/use-cases)[PrimitiveHow Taylor Hodge at Primitive uses Jetty runbooks that launch runbooks to test one plugin across every coding agent and model, and catch silent breaks early.](/guides/getting-ahead-of-new-model-updates-with-primitive)[Dial0How Dial0 runs two Jetty runbooks for an animated film: one turns a master recording into band sheet music, one grades every new render of a shot for drift.](/guides/from-the-first-note-to-the-final-frame-with-dial0)[Nous ResearchHow Morgane Moss at Nous Research used Jetty to run hundreds of long agent trajectories in parallel and optimize a skill through cheap, repeatable evals.](/guides/small-evals-beat-big-rewrites-with-nous-research)[ResearchPapers Jetty has contributed to, on agent benchmark standards, reproducible ML evaluations and AI accountability.](/guides/research) * * * ## Ready to build? **[Get started →](/docs/get-started)** — token to first runbook in under ten minutes. Prefer to talk it through? [Book a 20-minute walkthrough →](https://calendly.com/jon-jetty/jettyio-20-minute-onboarding) Bring a task from your own work and we'll turn it into a runbook together.