eve + Jetty

You shipped an eve agent, built on Vercel's filesystem-first framework where an agent is a directory. You tweaked a prompt. Is it better or worse? You can't tell from one reply. That's the run, check, fix, rerun loop from How to build an AI agent, and Jetty is the check.

eve owns the agent loop. Jetty grades each output and keeps it: every run becomes a trajectory you can score, label, and diff across versions, so a regression shows up before a customer finds it. The grader is a Jetty runbook the agent under test never sees, so a model can't grade its own output and rubber-stamp a regression.

The agent-side integration ships as @jetty/eve, an eve extension you mount in one file — live ingest of every turn, a grade-steered experiment bandit, and a scoreboard tool. The example is examples/eve-jetty in the SDK repo. It A/B-tests a help-desk triage agent in two configs (warm and terse), grading every reply and flagging the config that regressed:

TICKETS: 2   GRADER: rubric (independent)
 config        pass   avg   $/run
 v1 (warm)    2/2    4.7   0.0093  ✅
 v2 (terse)   0/2    2.7   0.0032  ❌  regressed
→ v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm).

eve already has evals. Where Jetty fits

eve ships a real eval system (defineEval, eve eval, an LLM-as-judge, a CI deploy gate). Those are your unit and integration tests: you author the assertions, they run your scripted sessions, they gate a deploy. Jetty sits beside them: an independent grader the builder didn't write, and a durable store where every run is a labelled trajectory you can diff across versions, models, and many agents, long after the CI job ended. Keep eve eval for “did this commit break a rule”; add Jetty for “which version is better, and is it improving over time.”

Install

Jetty plugs into eve two ways: @jetty/eve, an eve extension the agent mounts (live grading, an experiment bandit, an eval reporter), and the plain SDK for harness-side orchestration:

npm install @jetty/sdk @jetty/eve

Requires @jetty/sdk 0.2.0+ (for gradeWithJetty) and, for the @jetty/eve extension, eve 0.25+ on Node 24+ (the built-distribution extension model — a compiled dist/ plus a compatibility manifest — landed in eve 0.25.0).

Mount the extension

An eve extension is an agent-shaped package mounted under agent/extensions/; its contributions compose into the agent under the mount's namespace. The example's entire agent-side integration is one mount file (agent/extensions/jetty.ts):

// agent/extensions/jetty.ts — the file name is the namespace
import jetty from "@jetty/eve";

export default jetty({
  collection: process.env.JETTY_COLLECTION ?? "",   // empty → extension no-ops
  task: "triage-live",
  judgeMode: "simple_judge",             // grade inline with a native Jetty judge
  arms: { warm: "…", terse: "…" },       // your reply styles; the bandit is included
  contract: "Respond with ONLY the triage JSON object { … }",
});

What the mount contributes, under the jetty namespace:

Config is validated (zod) and binds once at the mount site; with no JETTY_COLLECTION everything degrades gracefully, so the agent runs without Jetty credentials. Source: packages/eve in the SDK repo.

The grading flow

eve's agent is a directory you run with npx eve dev (or deploy to Vercel). Drive it over the typed eve/client, then hand the reply to a Jetty grading task and wait for the trajectory. The grade comes back as a row you can label and compare:

import { Client } from "eve/client";
import { JettyClient, gradeWithJetty } from "@jetty/sdk";

const eve = new Client({ host: process.env.EVE_URL ?? "http://127.0.0.1:2000" });
const jetty = new JettyClient(); // JETTY_API_TOKEN from env or ~/.config/jetty/token

// 1. eve runs the agent (it owns the loop).
const turn = await (await eve.session().send(prompt)).result();

// 2. Jetty grades it server-side, with a grader that isn't the author —
//    upload, run the grader, read the grade, and label, in one call.
const { grade, trajectoryId } = await gradeWithJetty(jetty, "acme", "triage-grader", {
  files: [{ filename: "case.json", data: turn.message ?? "" }],
  labels: (g) => ({ "eval.grade": String(g.total) }), // labels can read the grade
});

Each grade is a Jetty trajectory: the inputs, outputs, grade, and cost, ready to replay. Compare the eval.* labels across configs to see which version slipped.

Native reporter. For eve's own evals, the Jetty() eval reporter drops into evals.config.ts exactly where the built-in Braintrust(...) reporter goes, so every eve eval result lands in Jetty automatically: import { Jetty } from "@jetty/eve/reporter" → defineEvalConfig({ reporters: [Jetty()] }). With no JETTY_COLLECTION set it no-ops, so the config is safe to commit.

Configure

Credentials. Put anything sensitive in secretParams, which the server keeps out of the stored trajectory. Don't put secrets in initParams; that field is persisted. The SDK never logs your token. Tokens resolve from a constructor arg, then JETTY_API_TOKEN, then ~/.config/jetty/token.

What Jetty captures

eve reports token usage but no dollar cost, so the example estimates $/run from tokens and a small per-model price table (src/cost.ts). Tune it to your real rates.

Full walkthrough: catch a regression

A step-by-step run of the example: first offline (no keys, ~10 seconds), then live. There are three levels of do I need a key?:

1. Offline demo (npm run demo): no keys, no eve, no network.
2. Grading on Jetty: covered by the free trial (10 runs, auto-activated). No API key.
3. The live eve agent runs via npx eve dev and resolves models through AI Gateway, so it uses your AI Gateway credential.

1. Clone and build

Run the example bundled in the repo. The -w @jetty/sdk flags refer to this monorepo's workspaces, so clone it and run everything from inside the checkout. (To use the SDK in your own app instead, jump to Use the SDK in your own project.)

git clone https://github.com/jettyio/jetty-sdk.git
cd jetty-sdk
npm install                    # installs the SDK, the example, and eve (one workspace install)
npm run build -w @jetty/sdk    # the example imports the built SDK
cd examples/eve-jetty

2. Run the offline demo (no keys)

npm run demo

You should see the verdict table immediately:

Acme Helpdesk — did my last change to the triage agent make it worse?
(simulated; run `npm run ab-eval` against a live `npx eve dev` for the real thing)

TICKETS: 5   GRADER: rubric (independent)

 config        pass   avg   $/run
 ------------  -----  ----  -------
 v1 (warm)    5/5    4.5   0.0051  ✅
 v2 (terse)   1/5    3.5   0.0039  ❌  regressed

→ v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm).

It also writes report.html and opens it: a styled verdict and per-run breakdown, the same report the live run produces. This is a deterministic stand-in, no spend. If you only want to understand the example, you can stop here.

3. Configure credentials (for the live run)

cp .env.example .env

Edit .env:

AI_GATEWAY_API_KEY=...                 # or VERCEL_OIDC_TOKEN, for the eve agent
JETTY_API_TOKEN=mlc_...                # your Jetty token
JETTY_COLLECTION=your-collection       # a collection your token can write to
JETTY_GRADE_TASK=triage-grader         # leave as-is
EVE_URL=http://127.0.0.1:2000          # where `npx eve dev` serves the agent

Load it into your shell (the scripts read process.env; they don't auto-load .env):

set -a && . ./.env && set +a

4. Deploy the grader (one time)

The harness calls a Jetty runbook that scores each draft. Deploy it into your collection:

npm run deploy-grader

This creates the triage-grader task from grader/RUNBOOK.md (and pushes a provider key into the collection if you have one). Re-running it updates the task.

5. Serve the agent and run the live A/B

# terminal 1: serve the eve agent (needs Node 24+)
npx eve dev

# terminal 2: A/B-eval it (start small: each ticket is a real server-side grade)
EVAL_TICKETS=2 npm run ab-eval

You'll see a line per run, then the verdict table:

  v1 (warm) · reset: 4.7 PASS
  v1 (warm) · double-charge: 4.7 PASS
  v2 (terse) · reset: 2.7 fail
  v2 (terse) · double-charge: 2.7 fail

TICKETS: 2   GRADER: rubric (independent)
 config        pass   avg   $/run
 v1 (warm)    2/2    4.7   0.0093  ✅
 v2 (terse)   0/2    2.7   0.0032  ❌  regressed
→ v2 (terse) is cheaper but fails the bar (4.0). Keep v1 (warm).

…then it writes report.html and opens it: the verdict, a per-run breakdown, and links to each Jetty trajectory. The eval caught that the terse config regressed, before it ever reached a customer.

Run it on Jetty's free trial (no key)

Jetty gives every collection a free trial (10 runs, auto-activated, on Jetty's keys). When the collection has no provider key, grading uses the trial automatically — no flag, no key-push:

EVAL_TICKETS=2 npm run ab-eval

The trial covers server-side Jetty runs (the grader). The eve agent runs on your machine via npx eve dev, so the full live run still needs an AI Gateway credential for the agent. But you can see Jetty's grading and trajectories on the trial, and npm run demo needs no keys at all. (Sonnet and most models are covered; Opus-class is excluded. After 10 runs, add your own key in Settings.)

6. Inspect what got stored

Every grade is a Jetty trajectory, labelled with eval.config, eval.grade, eval.pass, and eval.cost_est_usd. View them in the Jetty UI (https://flows.jetty.io/<collection>/triage-grader) or from code:

import { JettyClient } from "@jetty/sdk";
const jetty = new JettyClient(); // reads JETTY_API_TOKEN
const list = await jetty.listTrajectories(process.env.JETTY_COLLECTION!, "triage-grader", 5);
for (const t of list.trajectories) {
  const full = await jetty.getTrajectory(process.env.JETTY_COLLECTION!, "triage-grader", t.trajectory_id);
  const labels = Object.fromEntries(full.labels.map((l) => [l.key, l.value]));
  console.log(t.trajectory_id, labels["eval.config"], labels["eval.grade"], labels["eval.pass"]);
}

Because the runs are durable and labelled, you can compare configs across releases, long after the terminal session that produced them.

How it works (the pieces)

The SDK does the orchestration: runWithFiles/runAndWait (with file upload), getTrajectory, downloadFile, addLabel, createTask. That's the part worth copying into your own eval.

Use the SDK in your own project

Everything above runs the example inside this repo. To use the SDK in a new, standalone project, you don't need this repo or any workspaces. Install the published package from npm:

mkdir my-app && cd my-app
npm init -y
npm pkg set type=module          # the SDK is ESM
npm install @jetty/sdk @jetty/eve eve

The pattern isn't eve-specific: anywhere you can produce an agent output and call jetty.runAndWait(...) on it (eve, Flue, LangChain, a raw provider SDK, a hand-rolled loop), Jetty drops in as the eval layer. Copy the orchestration from src/ab-eval.ts into your own code.

Make it yours

• Add cases: append to TICKETS in src/tickets.ts.
• Compare your own versions: edit the two entries in CONFIGS, or change EVE_MODEL in agent/agent.ts to A/B models.
• Change the live arms: the styles the bandit routes between are the arms config at the mount site (agent/extensions/jetty.ts) — add a fourth, reword one, or drop to two.
• Move the bar: change PASS_BAR in src/eval.ts (batch A/B) or passBar at the mount site (live judge).
• Swap the grader: the rubric in grader/RUNBOOK.md is plain Python. Replace it with an LLM-judge call for model-based grading, then npm run deploy-grader.
• Override a contribution: turn the mount into a directory (agent/extensions/jetty/extension.ts + an override slot) to approval-gate or replace the extension's tool, hook, or resolver.

Protect sensitive content

Trajectories persist step inputs and outputs, so they're content-bearing. Put credentials in secretParams (kept out of the stored trajectory), not initParams. If a draft can carry PII, redact it before grading, or grade a hash or summary instead. Treat trajectory storage like any other logging surface.

Troubleshooting

• The harness can't reach the agent. Start it first with npx eve dev (Node 24+) and point EVE_URL at it. eve's HTTP channel fails closed for non-loopback traffic; for a deployed agent, add an authenticator in agent/channels/eve.ts.
• The agent didn't return JSON. The triage prompt asks for a bare JSON object; extractTriage tolerates fences and prose, but a chatty model can still wander. Tighten agent/instructions.md.
• grader produced no /app/results files. The grader must run on a supported model, keep its secrets frontmatter, and write to /app/results/. All three are set in grader/RUNBOOK.md.
• The live run is slow. Each grade spins up a sandbox (a few minutes for 2 tickets); that's expected, and the offline demo (npm run demo) is the fast path.

Two complementary paths: the batch A/B grades through a Jetty task (Jetty runs the rubric, which creates the trajectory) — Jetty as the independent grader. The @jetty/eve extension and the Jetty() eval reporter instead push finished eve turns and eval results in via the trajectory-ingestion endpoint — Jetty as the durable scoreboard. The example uses both.

See also

• The worked example · step-by-step tutorial on GitHub
• @jetty/sdk on npm · SDK API reference
• eve docs · Flue + Jetty · Bring your own framework
• How to build an AI agent