Get Started
Documentation menu

Defining Agent Checklists

A big part of investigating the quality of an AI workload is understanding how its evaluations performed over a period of time. This guide covers one of the most powerful tools you can define to get quick gains in an AI workload: the agent checklist.

What is an AI agent checklist?

Checklists in Jetty describe conditions that determine completion of the AI workload. Each item is something the agent can verify for itself before it stops. For example: a CSV has one row per input image, or the summary names every flagged receipt. The agent ticks through the list, fixes what fails, rechecks, and only then writes a report with an overall pass or fail.

The checklist is one of three instruments Jetty uses to judge a run. Code checks fail a run outright, checklist items keep the agent on track while it works, and judges score what passed after the run.

job donejob recordedRUNNING JETTY.IO SANDBOXINVESTIGATION1 · Code checkscode decidesWHENduring the runVERDICTpass or failON FAILthe run is marked FailedCATCHESwhat code can assert2 · Checklistthe agent confirmsWHENduring the run, before it stopsVERDICTconfirmed, item by itemON FAILfix & recheck, 3 rounds at mostCATCHESwhat you already know goes wrong3 · Judgesa model scoresWHENafter the run, on stored outputsVERDICT1–5, with a reasonON FAILmarked Needs workCATCHESwhat you can describe, not assertturn judgement into online checksspeed up checks with code

Closing the loop before you ship

“It works on my machine” is a classic trope in software engineering. We’ve spent two decades moving away from a laptop-centric testing model and these days agents are bringing us back to this mode of operation: running agent loops for hours and hoping that the laptop doesn’t die between 5 PM and 9 AM.

A checklist is a key part of keeping the agent on task and focused. Without a clear definition of done, and acceptance criteria encoded as a list of requirements at the end of the workload, agents will often arrive at open-ended results and unpredictable outcomes.

How do you take an open agent loop and close it around a well-defined outcome? Here are some steps to get started:

  1. Set an objective. Not “does it respond”, but what a correct, complete answer looks like, and what an unacceptable one looks like. If you can't describe the difference, you can't evaluate it, and neither can anyone else on your team.
  2. Turn experience into guards against common failures. Check for hallucinated tool calls, half-followed instructions, early exits, and endless loops. Every AI workload accumulates its own history of failures, and ready to become another guard in a future checklist.
  3. Run an investigation and measure quality. Assemble representative tasks, including hard or adversarial ones, so you can see which model and runtime configurations perform best.

The checklist is part of every agent run, so you can use the results as part of deciding whether the AI workload is performing correctly.

Running an agent investigation

Investigations in Jetty are framed around a simple question. For instance, “which model is cheaper for this task?” or “which tasks performed the best/worst and why?”. Investigations allow you to retrospectively analyze your AI workload performance. Key facts about investigations:

  • They run in parallel over a batch. A single run tells you what happened once. Dozens of runs across your eval suite tell you what happens generally.
  • They judge quality, not just completion. Some tasks need more powerful evaluations, such as LLM judges and rubrics. Tests are boolean, evals are a gradient explains how to pick the right judgment for each part of the task.
  • They can measure across models, agents, MCP servers, and runtime configuration. If your whole setup only works with one specific model, you don't have a robust agent, you have a fragile one that happens to work today. Jetty is model-agnostic, so one eval suite covers every configuration, and every run is recorded so you can reconstruct how an outcome occurred instead of guessing.
  • They help maintain eval standards over time. When you change the task, the tools, or the target behavior, your eval suite needs to change with it, and you need to know which version of the suite produced which results. Improving agent runbooks covers versioning both together.

When your investigations start to yield insights into how your AI workload is performing, you're ready to improve performance in production.

Improving performance in production

In production, AI workloads are typically triggered by an API call, an MCP server, or another agent. Four strategies keep agents honest once they are there:

  • Run investigations after every meaningful change. Whenever a new model version drops, or a new condition is required, use the evaluations to benchmark the change before it reaches production.
  • Schedule the workload so evaluations run periodically and you are notified automatically.
  • Watch for drift, not just outright failure. An agent can stay technically functional while quietly getting worse at the parts of the job that matter most. We discuss this in our article on valley-dodging.
  • Revisit your baseline periodically. What counted as “good enough” six months ago might not anymore, especially if the models underneath your agent have changed. Update the bar on purpose instead of letting it drift on its own.

Where Jetty fits

Maintaining an agent checklist over several weeks is a challenge for any team with a growing number of AI workloads. That management challenge is exactly what Jetty is built to handle: the parts that don't scale are tracked and maintained automatically. You bring your own agent and model over MCP, run your eval suite against it, and Jetty tracks the rest underneath, the same bring-the-job, we-handle-the-infra approach behind everything Jetty runs.

Human judgement is more important now than ever, and agent checklists are a simple and effective method of embedding expectations into long agent workloads. Verifiable agents improve trust in complex systems, and a checklist recorded on every run is what makes an outcome traceable.

Run the checklist

If you've been meaning to put your agent through a real eval process and haven't gotten past step one, that's normal, and it's exactly what Jetty is for. Sign up to get a free Jetty account and run your first checklist against real data today.