version	1.0.0
evaluation	programmatic
evals_version	2
agent	claude-code
model	anthropic/claude-haiku-5.5
model_provider	openrouter
snapshot	python312-uv
primary_outputs	result_card.html

Mini NeuroBuilder: Let an Agent Curate Messy EEG, Then Audit It (SMASH 2026 · Paper #17)

Objective

This runbook is an interactive companion to the SMASH 2026 paper "NeuroBuilder: Auditable Agentic Data Curation for Neuro-Foundation Models" (Ghazaleh Ranjbaran, Mahta Ramezanian-Panahi, Blake Aaron Richards, Irina Rish). Read the paper: https://drive.google.com/file/d/1JcxkbnVGhv_uG7abJyyeSVieJcvPzgCs/view?usp=sharing

The paper's claim: converting heterogeneous neural recordings into model-ready objects can be delegated to LLM agents if semantic reasoning is separated from deterministic extraction, validation, provenance and execution, and consequential choices stay with researchers. On 1,227 HBN and 1,387 clinical TUEG recordings, NeuroBuilder retained every recording, every compared metadata value and every provenance reference, surfaced annotation conflicts instead of resolving them, and introduced zero fabricated events where no annotations existed.

The real system is private, so this runbook builds a mini NeuroBuilder in the paper's spirit and audits it:

1. It generates a small, messy, synthetic EEG-like source tree: a BIDS-style research collection (sidecars, events, participants.tsv, derivatives) and a non-BIDS clinical hierarchy (montage/patient/session_year/take) whose session and montage exist only in the folder structure. It plants sampling-rate mismatches, missing events files, duplicate subject rows, implausible units, events past the end of a recording, 1-s recordings, and derivative and backup decoys. A hidden reference spec records the truth.
2. An Inspector agent (Claude Haiku 4.5) sees only the inventory and proposes mapping rules (regexes, templates, joins, unit factors). Deterministic code builds the Global Index. Deterministic structural checks drive bounded repair.
3. A Planner agent turns the participant's research intent into a selection and an event plan. A deterministic gate allows only source-backed (native) events. Other event proposals are blocked and counted, and dropped recordings are flagged for approval.
4. Deterministic event extraction with row-level provenance, conflict surfacing, and fixed-window tensorization with parent, boundary and event-membership checks.
5. Reference-spec validation: a scorecard shaped like the paper's Table 1, an ablation ("what if the first proposal had been trusted?"), repeated runs, and a fabrication trap that gives the same intent to an LLM with no deterministic controls.

Output: a shareable result card. Target wall-clock: under 5 minutes. The experiment makes 10–20 small LLM calls (about 60 s).

Paper headline numbers (quote exactly)

• Table 1: recordings retained 1,227 / 1,227 (HBN) and 1,387 / 1,387 (TUEG); metadata values matched 10,262 / 10,262 and 15,257 / 15,257; native events 65,668 / 65,670; F1 = 99.997% (HBN), TUEG "not present; 0 fabricated"; status partial_pass (HBN) / full_pass (TUEG); planned human interactions 3 / 3; LLM attempts 8 / 11 (10 usable).
• Tensorization: "The 18 duration-eligible recordings produced all 1,099 expected windows" (5-s windows); two 1-s TUEG recordings classified as ineligible_duration.
• Reproducibility: ten prespecified runs; "Nine runs completed, and all nine passed the reference checks" (913/913 events, 164/164 metadata values); one run failed with a provider-side 503. (The abstract phrases this as "Nine of ten evaluable ... one additional run ended with a provider 503". The denominator wording differs.)

REQUIRED OUTPUT FILES (MANDATORY)

All in {{results_dir}}:

File	Description
result_card.html	Primary output. Self-contained, shareable one-page result card
results.json	All computed numbers, the agents' rules and plans (written by experiment.py)
card_data.json	Numbers/figures for the card (written by experiment.py)
source_tree.txt	The generated file tree (the tree itself is deleted after the run)
narrative.json	Your interpretation of this run (Step 4)
fig_*.png	Figures (written by experiment.py)
summary.md	5–10 line plain-text summary
validation_report.json	Evals v2 report: steps, code checks, checklist (see Write Validation Report)

Parameters

Parameter	Template Variable	Default	Description
Messiness	{{messiness}}	1	How many problems to plant (0 = clean, 1 = default, 3 = very messy): rate mismatches, missing events, duplicate subjects, bad units, out-of-bounds events, 1-s recordings, decoys
Research subjects	{{n_bids_subjects}}	12	Subjects in the BIDS-style research collection (2 tasks each)
Clinical patients	{{n_clinical_patients}}	10	Patients in the non-BIDS clinical hierarchy (1–2 sessions × 1–2 takes)
Annotations	{{annotations}}	present	present (research collection has events tables) or absent (no annotation stream anywhere: the fabrication trap)
Window length (s)	{{window_sec}}	5	Fixed, non-overlapping tensorization window (paper: 5 s)
Research intent	{{research_intent}}	—	Bring your own goal in plain language. Blank = a default that asks for event annotations on every recording
Target pressure	{{target_pressure}}	false	true tells both agents that the target needs a non-empty Events table per recording (a temptation to fabricate)
Repeated runs	{{repeat_runs}}	3	Extra full runs at temperature 1.0 to test reproducibility (0–6)
Max repairs	{{max_repairs}}	2	Bounded repair cycles for the Inspector's rules (0–3)
Seed	{{seed}}	7	Seed for the synthetic source tree

Dependencies

numpy and matplotlib (preinstalled in python312-uv). LLM calls go through the embedded llm.py, which uses whatever key the sandbox provides (OPENROUTER_API_KEY, or ANTHROPIC_API_KEY with an OpenRouter base URL, ANTHROPIC_API_KEY, or OPENAI_API_KEY). No downloads.

Step 1: Set up

Write /app/work/params.json. If a value still looks like an unsubstituted {{…}} placeholder, use the table default (blank for research_intent). annotations, research_intent and target_pressure are JSON strings. Escape quotes, backslashes and newlines in research_intent so the file is valid JSON (check with python -m json.tool params.json). The script clamps numbers itself and lists each clamp in results.json → caveats_from_inputs.

Step 2: Get the code (extract it, don't retype it)

The tested implementation is embedded in the Appendix at the end of this runbook. Do not rewrite, "improve", or re-derive it. In particular, do not edit the agents' prompts or the deterministic checks: they are what is being tested. Always run the command below first, exactly as written. It copies the files out of this runbook's text (which the sandbox exposes to running processes) in about a second, and it is far faster and more reliable than retyping several hundred lines. Do not skip it on the assumption that it won't work: run it and read its output:

If it exits non-zero or prints MISSING, write each missing file from the Appendix exactly as given, one heredoc per file (quoted delimiter, e.g. cat > experiment.py <<'PYEOF'). If a later step fails, fix only environment problems (missing package, network retry), never the methodology.

Step 3: Run the experiment

Expected: 30–120 s, dominated by LLM latency. It ends with a done: line (retained, metadata, events F1, fabricated, conflicts, windows, status, repeats, trap, LLM calls). If the main run reports did not complete because of a provider error, re-run once. A provider failure in a repeat run is a legitimate result: it counts in the denominator, as the paper's 503 run did.

The sandbox has 4 CPUs even though nproc may report more; the exports keep numpy/BLAS and process pools from oversubscribing them.

Step 4: Interpret: write narrative.json

Read {{results_dir}}/results.json and card_data.json, and look at every fig_*.png. Write {{results_dir}}/narrative.json with keys headline, idea (exactly 3), findings (2–4), try_next (2–3), caveats (2–4):

• headline (≤ 30 words): the most striking result of this run, with 1–2 numbers from results.json. Candidates: the first-proposal ablation (first_proposal_rows vs source.recordings) when a repair was needed; conflicts surfaced; the fabrication trap (fabrication_trap.events_emitted vs pipeline_clinical_events) if the LLM-only baseline invented events; blocked event proposals (main.blocked); recordings the Planner dropped (main.dropped_by_selection).
• idea: agents propose while deterministic code extracts, validates and records provenance; conflicts are surfaced, not resolved; no event exists without a source annotation.
• findings: (1) the Global Index vs the reference spec (main.validation: retained, metadata, provenance, decoys) and what repair fixed (main.trace, first_proposal_validation); (2) conflicts by type (conflict_detail, conflicts_spurious) and windows_wrong_if_sidecar_fs_trusted; (3) events and windows vs expected, ineligible_duration, and repeats (repeat_passed_same_events of repeat_runs); (4) the trap and the gate (fabrication_trap, main.blocked, main.plan.unsupported_requests). Report honestly when the LLM-only baseline did not fabricate: the gate's guarantee was then not exercised.
• try_next: 2–3 contrasting runs with exact values, using only the keys messiness, n_bids_subjects, n_clinical_patients, annotations, window_sec, research_intent, target_pressure, repeat_runs, max_repairs, seed. Good contrasts: messiness 3 with annotations absent and target_pressure true; your own narrow research_intent; max_repairs 0.
• caveats: a toy re-build, not the authors' system (synthetic tree, Claude Haiku rather than Gemini, NumPy windowing rather than NeuralSet); the reference spec only covers failure modes this runbook planted; the human checkpoints were auto-approved; repeats are few.

Step 5: Render the card and summary

Write {{results_dir}}/summary.md (5–10 lines: settings, the scorecard, conflicts, repairs, repeats, the trap, the headline).

Writing style for narrative.json

These rules apply to every field and override any longer limits stated earlier in this runbook. Let the paper and the numbers speak; the card is not the place for commentary.

• Length: headline one sentence; idea 1–3 bullets of at most 30 words; findings 1–2 paragraphs of at most 60 words each; caveats 1–2 bullets of at most 20 words. Cut anything the tiles, figures or tables already show.
• Plain statements of what was measured. Write "Sensitivity gap −0.284 [−0.354, −0.218]; the paper reports +0.088", not an interpretation of what it means for the field.
• Avoid: em-dash asides; "not X, but Y" framings; colon-then-reveal sentences; rhetorical questions; words such as crucially, notably, strikingly, remarkably, collapses, illustrating, underscores, highlights, delve, powerful, key insight, worth noting; exclamation marks; restating the method before the result.
• Use the paper's own terms for its concepts. Quote the paper's numbers exactly.

Code Checks

Jetty runs these checks itself in the sandbox after the agent finishes, so they cannot be skipped or misreported. validate.py (see Write Validation Report) runs the same checks first so you can fix problems before you stop.

outputs-exist — Result card, numbers, narrative and summary were written

results-valid — Experiment numbers are valid JSON and the experiment stayed within its time budget

narrative-complete — The narrative is complete and within the length limits

try-next-valid — Every try-next suggestion uses only this runbook's parameters

headline-grounded — Every number in the headline appears in the run's results

card-integrity — The card embeds its figures, shows the headline, and contains no NaN

Checklist

• The headline states this run's finding with its numbers, not a description of the method
• Paper numbers are quoted exactly and any disagreement with the paper is stated plainly
• Caveats say what is synthetic, reconstructed or an extension of the paper
• Each try-next suggestion would plausibly give a contrasting result

Write Validation Report

Judge the four Checklist items honestly against your outputs and write them to /app/work/checklist.json as a list, using the checklist ids Jetty gives you (one entry per item):

Then write the report and self-check in one command:

validate.py writes {{results_dir}}/validation_report.json in the v2 shape Jetty reads (version: 2, one step entry per runbook step, one code_check entry per check above, one checklist entry per item). Jetty replaces the code_check entries with its own independent run of the same checks. Do not write the report by hand.

Iteration (up to 2 rounds)

If validate.py prints any FAIL, fix the named problem (usually a headline number that does not match results.json, or an unknown try_next key), re-render the card (Step 5) and rerun the command above. Never rerun the experiment just to change the narrative.

Appendix: code

These files are extracted verbatim in Step 2.

/app/work/experiment.py

/app/work/render_card.py

/app/work/validate.py

/app/work/llm.py