Evals as branding: from the first note to the final frame

01 · Situation

Dial0 needs to analyze and generate all kinds of content. For Radical Vermont, that meant two things: sheet music so a live band can perform the film’s score at the premiere, and evaluation metrics to catch AI drift and keep every shot consistent.

• Sheet music for a live band: Code checks verify the song files against their recorded hashes. Gemini listens to the recording to pick out the instruments, then music21 turns the MIDI into notes on the score, and the result is judged based on five criteria until it clears the quality bar.
• Proof that each shot is getting better: Code checks confirm both renders match, then Gemini acts as a judge, grading the new render against the last one from −5 to +5 on quality and checking seven continuity categories for drift. Independent checks combine those grades into a final verdict so the model can’t grade itself into a pass.

Dial0 turned both jobs into runbooks with Jetty’s coding agent plugin, so their own coding agent could write, run and refine their runbooks themselves.

02 · Solution

Both workloads run as Jetty runbooks with two ways in: either as a tool called by an agent, or the app triggers the run automatically from inside the Dial0 interface.

That’s the same bring-the-job, we-handle-the-infra model Jetty runs on generally: describe the job once, point it at your own agent, and Jetty tracks the run underneath it.

• Phonograph: song to sheet music, with a master recording in and an editable draft score and instrument parts out.
• Cinescope: render progress review, grading every new render of a shot against the last one.

03 · Outcome

A small independent team can now produce band-ready sheet music and keep every shot consistent. Both run as repeatable jobs on Jetty, started from the app and checked on later. Image evaluations are tracked, so models and providers can be compared, and time and budget are saved by avoiding repeated failures and speeding up improvements.