version	1.0.0
evaluation	programmatic
evals_version	2
agent	claude-code
model	anthropic/claude-haiku-5.5
model_provider	openrouter
snapshot	python312-uv
primary_outputs	result_card.html

Will This Eye Gain 10 Letters? A Glass-Box nAMD Prognosis You Can Read (SMASH 2026 · Paper #5)

Objective

This runbook is an interactive companion to the SMASH 2026 paper "Visual Acuity Improvement Prediction in Neovascular Age-Related Macular Degeneration" (Hadi Moazen, Mélanie Hébert, Alaa Eddine Ayadi, Ali Dirani, Audrey Durand). Read the paper: https://drive.google.com/file/d/1h81k0dJvGwdnHFxP-kYIcjuwFZgX_n1d/view

The paper's claim: instead of an end-to-end deep network on OCT scans, extract clinical OCT biomarkers automatically with deep learning (CST, SRF, IRF/IRC, PED, VMT), add baseline best-corrected visual acuity (BCVA), and feed them to an Explainable Boosting Machine (EBM), a glass-box additive model. This predicts a ≥ 10-letter gain one year after anti-VEGF loading better than end-to-end ResNet18 (AUC 0.77 vs 0.57), matches an EBM trained on human-measured biomarkers, and lets physicians read exactly why each prediction was made.

The participant describes their own hypothetical patient (baseline BCVA, CST, and which fluid/structure biomarkers are present) and chooses how error-prone the automated extraction is. The run builds a simulated cohort matching the paper's reported marginals, fits the glass box (EBM) on measured and on automated (noisy) biomarkers plus a gradient-boosting black box with 5-fold CV, then explains the participant's patient term by term and shows how much the prediction moves if the automated pipeline misreads one biomarker. Output: a shareable result card.

Target wall-clock: under 4 minutes. The experiment takes about 10–20 s.

The CHU de Québec cohort is private. Every patient in this run is simulated; effects are hand-specified and plausible, not estimated from the authors' data. Simulated data. Not for clinical use, not for counselling a real patient.

How the simulation is built

• Marginals from the paper (§2), n = 1203 eyes: BCVA 64 [53, 70] letters (quantile-matched), CST 289 [245, 349] µm (lognormal), SRF 43%, IRF 34%, PED 48%, VMT 17%; biomarkers mildly correlated with CST; IRF and thick CST go with worse BCVA.
• Outcome (assumed mechanism): 1-year letter gain = intercept + more room to gain at lower BCVA (plateau below 45 letters)
  • SRF (favourable) − IRF − PED − VMT + thicker CST (slightly favourable) + a CST × BCVA interaction + a 30% "strong responder" subgroup + noise, with a ceiling on achievable BCVA. The intercept is calibrated so 25% gain ≥ 10 letters.
• "Automated" biomarkers: each binary biomarker is flipped with probability (1 − the paper's extraction accuracy) × automation_error_scale (accuracies SRF 0.91, IRF/IRC 0.88, PED 0.89, VMT 0.82); CST gets Gaussian error cst_error_um.
• Models (paper §3–4): EBM (interpret-core, 5 pairwise interactions, 8 outer bags) on measured and on automated biomarkers; scikit-learn histogram gradient boosting as a stand-in for XGBoost; a BCVA-only logistic regression as a floor. 5-fold stratified CV; the decision threshold is the training prevalence (equivalent to class-weighting). If interpret cannot be installed, the script falls back to a transparent binned additive logistic model (the card then says "Additive GAM"). The end-to-end ResNet18 baseline needs OCT images and is not reproduced.

Paper headline numbers (quote exactly)

• Table 1 (≥ 10-letter gain at one year): ResNet18 AUC 0.57 ± 0.05 (sensitivity 0.15 ± 0.04); XGBoost 0.74 ± 0.06 (sens 0.73 ± 0.01, spec 0.63 ± 0.04, F1 0.53 ± 0.03); EBM 0.77 ± 0.07 (sens 0.75 ± 0.08, spec 0.61 ± 0.03, F1 0.54 ± 0.02, accuracy 0.65 ± 0.03).
• EBM on human-measured biomarkers: AUC = 0.77, the same as on automated biomarkers (Figure 2b).
• Biomarker extraction accuracy: SRF 0.91, IRF/IRC 0.88, PED 0.89, VMT 0.82; CST "average error of 0.1437 µm".
• Most important predictors: "baseline VA, CST, and the interaction between these two features"; improvement starts to decline above 50 letters, with a sharp drop above 75 (ceiling effect).
• Cohort: 1203 eyes; 299 (25%) gained ≥ 10 letters; median gain 2 [-3, 10]; BCVA at one year 68 [57, 76].
• Internal inconsistencies worth knowing: XGBoost AUC is 0.74 in Table 1 but 0.75 in the Figure 2a legend; the CST error of 0.1437 µm is implausibly small for OCT (the text also says CST is measured "in nanometers"), so this runbook uses a participant-set CST error (default 20 µm); "2.67 times more patients who did not see an improvement" implies ~27% improvers, vs 25% reported.

REQUIRED OUTPUT FILES (MANDATORY)

All in {{results_dir}}:

File	Description
result_card.html	Primary output. Self-contained, shareable one-page result card
results.json	All computed numbers (written by experiment.py)
card_data.json	Numbers/figures for the card (written by experiment.py)
narrative.json	Your interpretation of this run (Step 4)
fig_*.png	Figures (written by experiment.py)
summary.md	5–10 line plain-text summary
validation_report.json	Evals v2 report: steps, code checks, checklist (see Write Validation Report)

Parameters

Parameter	Template Variable	Default	Description
Patient: baseline BCVA	{{patient_bcva}}	50	Your hypothetical patient's baseline acuity in ETDRS letters (cohort median 64)
Patient: CST	{{patient_cst}}	380	Central subfield thickness in µm (cohort median 289)
Patient: SRF	{{patient_srf}}	1	Subretinal fluid present (1) or absent (0)
Patient: IRF/IRC	{{patient_irf}}	1	Intraretinal fluid / cysts present (1) or absent (0)
Patient: PED	{{patient_ped}}	0	Pigment epithelial detachment present (1) or absent (0)
Patient: VMT	{{patient_vmt}}	0	Vitreomacular adhesion/traction present (1) or absent (0)
Automation error scale	{{automation_error_scale}}	1	Multiplies the paper's biomarker error rates (0 = perfect extraction, 1 = paper, 3 = much worse)
CST error	{{cst_error_um}}	20	SD of the automated CST measurement error in µm
Cohort size	{{n_patients}}	1203	Simulated eyes (paper: 1203)
Seed	{{seed}}	5	Random seed

Dependencies

numpy, scikit-learn, matplotlib (preinstalled in python312-uv) plus interpret-core (the EBM; one ~15 MB wheel). No data downloads.

Step 1: Set up

Write /app/work/params.json (JSON numbers; if a value still looks like an unsubstituted {{…}} placeholder use the table default):

Validate before running: clamp patient_bcva to [5, 88], patient_cst to [150, 700]; biomarkers must be 0 or 1 (round anything else); clamp automation_error_scale to [0, 4], cst_error_um to [0, 150], n_patients to [300, 5000]. Record any correction as a caveat.

Step 2: Get the code (extract it, don't retype it)

The tested implementation of the method is embedded in the Appendix at the end of this runbook. Do not rewrite, "improve", or re-derive it. Always run the command below first, exactly as written. It copies the files out of this runbook's text (which the sandbox exposes to running processes) in about a second, and it is far faster and more reliable than retyping several hundred lines. Do not skip it on the assumption that it won't work: run it and read its output.

If it exits non-zero or prints MISSING, write each missing file from the Appendix exactly as given, one heredoc per file (quoted delimiter, e.g. cat > experiment.py <<'PYEOF'). If a later step fails, fix only environment problems (missing package, network retry), never the methodology.

Step 3: Run the experiment

Expected: 10–20 s. Ends with a done: line (patient probability, the four AUCs, the most damaging misread).

The sandbox has 4 CPUs even though nproc may report more; the exports keep numpy/BLAS and process pools from oversubscribing them.

Step 4: Interpret: write narrative.json

Read {{results_dir}}/results.json and card_data.json, and look at every fig_*.png. Write {{results_dir}}/narrative.json with keys headline, idea (exactly 3), findings (2–4), try_next (2–3), caveats (2–4):

• headline (≤ 30 words): this run's most striking result with 1–2 numbers copied from results.json, usually the patient's predicted chance (patient.p_gain_10plus, as a %) and how far one misread biomarker moves it (patient.worst_misread). Say "simulated" in the headline.
• idea: (1) extract interpretable OCT biomarkers automatically instead of predicting end-to-end from images; (2) a glass-box additive model (EBM) keeps the physician able to check every term; (3) automated biomarkers predict as well as human-measured ones in the paper.
• findings: (1) the patient: probability, the largest contributions from patient.contributions (logits; positive = towards a ≥ 10-letter gain) and what they mean clinically (e.g. lower baseline BCVA = more room to gain); similar_observed_rate if not null; (2) misreads: which biomarker error moves the prediction most and by how much; (3) the models: glass box on automated vs measured vs gradient boosting vs BCVA alone (models, AUC mean ± SD) vs the paper's Table 1; note when the margin over BCVA-alone is small; (4) the BCVA shape function vs the paper's Figure 3b. Be explicit that the effects were specified by the simulation, so the model "recovering" them is a demonstration of interpretability, not evidence about real patients.
• try_next: 2–3 contrasting runs with exact values, using only the keys patient_bcva, patient_cst, patient_srf, patient_irf, patient_ped, patient_vmt, automation_error_scale, cst_error_um, n_patients, seed. Good contrasts: a patient near the ceiling (patient_bcva 78); a much worse extraction pipeline (automation_error_scale 3); perfect extraction (automation_error_scale 0, cst_error_um 0).
• caveats (2–4): simulated cohort with hand-specified effects (only the marginals come from the paper); not for clinical use; the end-to-end ResNet18 and the real DL extraction pipeline are not run (extraction errors are simulated from the reported accuracies); the CST error is the participant's setting because the paper's 0.1437 µm looks like a units slip.

Step 5: Render the card and summary

Write {{results_dir}}/summary.md (5–10 lines: patient, predicted chance, top contributions, worst misread, AUC table, headline, "simulated, not for clinical use").

Writing style for narrative.json

These rules apply to every field and override any longer limits stated earlier in this runbook. Let the paper and the numbers speak; the card is not the place for commentary.

• Length: headline one sentence; idea 1–3 bullets of at most 30 words; findings 1–2 paragraphs of at most 60 words each; caveats 1–2 bullets of at most 20 words. Cut anything the tiles, figures or tables already show.
• Plain statements of what was measured. Write "Sensitivity gap −0.284 [−0.354, −0.218]; the paper reports +0.088", not an interpretation of what it means for the field.
• Avoid: em-dash asides; "not X, but Y" framings; colon-then-reveal sentences; rhetorical questions; words such as crucially, notably, strikingly, remarkably, collapses, illustrating, underscores, highlights, delve, powerful, key insight, worth noting; exclamation marks; restating the method before the result.
• Use the paper's own terms for its concepts. Quote the paper's numbers exactly.

Code Checks

Jetty runs these checks itself in the sandbox after the agent finishes, so they cannot be skipped or misreported. validate.py (see Write Validation Report) runs the same checks first so you can fix problems before you stop.

outputs-exist — Result card, numbers, narrative and summary were written

results-valid — Experiment numbers are valid JSON and the experiment stayed within its time budget

narrative-complete — The narrative is complete and within the length limits

try-next-valid — Every try-next suggestion uses only this runbook's parameters

headline-grounded — Every number in the headline appears in the run's results

card-integrity — The card embeds its figures, shows the headline, and contains no NaN

Checklist

• The headline states this run's finding with its numbers, not a description of the method
• Paper numbers are quoted exactly and any disagreement with the paper is stated plainly
• Caveats say what is synthetic, reconstructed or an extension of the paper
• Each try-next suggestion would plausibly give a contrasting result

Write Validation Report

Judge the four Checklist items honestly against your outputs and write them to /app/work/checklist.json as a list, using the checklist ids Jetty gives you (one entry per item):

Then write the report and self-check in one command:

validate.py writes {{results_dir}}/validation_report.json in the v2 shape Jetty reads (version: 2, one step entry per runbook step, one code_check entry per check above, one checklist entry per item). Jetty replaces the code_check entries with its own independent run of the same checks. Do not write the report by hand.

Iteration (up to 2 rounds)

If validate.py prints any FAIL, fix the named problem (usually a headline number that does not match results.json, or an unknown try_next key), re-render the card (Step 5) and rerun the command above. Never rerun the experiment just to change the narrative.

Appendix: code

These files are extracted verbatim in Step 2.

/app/work/experiment.py

/app/work/render_card.py

/app/work/validate.py