Building AI Evaluations

Story time: one of the big sources of AI quality comes from reading social media posts when a new model drops and figuring out if the “vibes” are good. This is the state of AI evaluation at the frontier.

We can do better; instead of waiting for social proof from social media, you can use agent transcripts to evaluations that form part of your AI workload to avoid the negative chatter in the first place. The key is selecting an agent evaluation tool that continually improves your production AI workload.

Jetty is an AI evaluation tool that gives you a way to know, before you ship, whether your agent is actually doing the job it was built to handle at the level of quality you expect.

You need a tool that’s going to tell you quickly where to focus your effort. To do that each run needs a score that you can quickly pull up so you can track defects and surface errors. From there, you launch an investigation–a deep dive into what really happened. Sometimes this means re-running the same thing under a different runtime configuration and measuring the change. Other times, this might entail changing your evaluations to shine a stronger light on where the agent is tripping up. This process is exactly what a good agent evaluation tool enables.

Since running investigations is itself going to involve more runs, you need a suite of tools that can track these changes without you having to get involved in maintaining metadata.

Marie Kondo Your Evals

Do these AI evals spark joy? That’s the right question to be asking yourself when designing the agent evaluations. An agent can solve a task in a dozen valid ways, or fail in a dozen invalid ones, and a simple pass or fail check can't tell the difference. We talk about the swiss-cheese approach to testing in tests are boolean, evals are a gradient: you need a diversity of evaluations to gain sufficient intuition into where an AI workload is struggling.

Evaluating an agent means asking a bigger question on every run: did it use the tools correctly, did it reason through the problem soundly, would a person actually accept what it produced? That judgment doesn't come from one test. It comes from running the agent many times, on many inputs, and looking at the pattern of what breaks and what doesn't.

Just like the things we accumulate in life, to reach serenity, we need to let go. Some AI evaluations need to be discarded when they are superseded. For instance, if your AI workload is effectively extracting the text, the checks around this could be added noise when trying to parse dates correctly post-extraction.

The many sources of joy from your AI evaluation

The 5 Category checklist that matters.

• Task completion. Did the agent finish the job with the outputs we expect?
• Tool and skill use. If the agent has access to tools, like a skill, MCP server, is it using them the way you intended? If they are working around them, it will show up in Jetty’s agent logs.
• Failure modes, not just failures. A run that fails tells you something happened, see investigating agent quality, which discusses how to create meaningful analysis.
• Judgment, not just scoring. Use a mix of code checks, checklists and judges. Judges, particular, are a great way to direct further improvements. And a good eval setup will avoid overlap between evaluation tools.
• Scale. One good run is a great start, but thousands of runs across a real range of inputs will really inform whether the agent is actually reliable.

Imagine your ideal lifestyle AI Workload

The pattern that works, whether you build it yourself or use a tool, is the same loop every time: run the agent, see where it breaks, investigate, improve and run again. To do this quickly, launch runs in parallel that will form a benchmark of realistic inputs and outputs. This is your team’s insurance policy for when something looks off two weeks from now.

Close Your Laptop and Start Your Journey

This is the loop Jetty is built around. You bring your own agent and model, and Jetty handles the infrastructure underneath: running long runs in parallel, storing every runtime configuration automatically, and surfacing the failure modes from each run so you know what to fix next. That's the same bring-the-job, we-handle-the-infra model behind everything Jetty runs, which is why agent evaluations sit at the center of the product instead of bolted on the side.

Avoid being locked into one model or one harness. Teams using Jetty describe this way of working as being closer to the KonMari Method: regularly letting go of outdated evaluations and adopting improvements to the AI workload in light of experience. Start your own journey: close your laptop, and come back to something you can act on instead of a pile you have to sift through.

Start Komono Evaluation

If your agent has never been through a real eval loop, the fastest way to organize and categorize your evaluations is to run it with Jetty. Sign up to get a free Jetty account and run your first batch of evals today.