version	1.0.0
evaluation	programmatic
evals_version	2
agent	claude-code
model	anthropic/claude-haiku-5.5
model_provider	openrouter
snapshot	python312-uv
primary_outputs	result_card.html

Gap Audit: Is That Subgroup Gap Real? (SMASH 2026 · Paper #7)

Objective

This runbook is an interactive companion to the SMASH 2026 paper "Prevalence, Thresholds, and Subgroup Performance Gaps in Clinical Prediction Models Across Five Public Datasets" (Zavash Ghorbanpourbaboli, Arina Afshar, Elham Jangravi, Mohammed Ayoub Alaoui Mhamdi, Russell Butler). Read the paper: https://drive.google.com/file/d/1MnK-mFUFDe2Josf706Rgn4RxZMB6df3M/view?usp=sharing

The paper's claim: a raw subgroup performance gap is not a fixed property of a model. Its size and even its sign depend on the metric, outcome prevalence, operating threshold, sample composition and uncertainty, so a gap is a signal to investigate, not a verdict of (un)fairness.

The participant picks a public dataset and a demographic comparison (or pastes their own model's confusion matrices). The run reproduces the paper's workflow end-to-end and produces a shareable result card showing how much of the gap survives:

1. Uncertainty: sensitivity difference with a subgroup-stratified bootstrap CI (paper's primary endpoint)
2. Prevalence: PPV gap before vs after standardizing to a common prevalence
3. Threshold: every metric's gap swept across operating thresholds
4. Analyst choices: a "multiverse" of defensible data-preparation choices (an extension that makes the paper's sample composition point visible)

Target wall-clock: under 5 minutes. Do not explore beyond these steps.

The method, as in the published paper (§2 and predictor table)

• Restrict to the two compared groups; exclude the demographic variable from predictors. Predictors follow the paper's table (CDC: the sex comparison keeps Age, the age comparison drops Age and Sex; Diabetes-130: counts, admission codes, the 23 medication columns, max_glu_serum, A1Cresult, change, diabetesMed, plus age_numeric and race indicators unless they define the comparison).
• 70/30 train/test split, seed 42, separately per comparison. Diabetes-130 keeps every encounter and partitions, cross-validates, calibrates and resamples by patient; its 0–9 age band is excluded from the age comparison.
• One shared model per comparison, selected among logistic regression, L1 logistic regression, random forest, shallow tree, gradient boosting and a soft-voting ensemble by 5-fold training-only CV AUC; then 5-fold isotonic fold-ensemble calibration on training data.
• Threshold τ = unweighted training-sample outcome prevalence, applied to both groups.
• Endpoint: sensitivity difference (Male − Female, youngest − oldest, first-named race group − reference). Age bands: CDC 18–44 vs 65+; Diabetes-130 10–29 vs 60+.
• 95% percentile CI from 1,000 replicates: rows resampled within each subgroup; for Diabetes-130, patients resampled with replacement, each carrying all of their encounters.
• PPV standardization: hold each group's Se and Sp fixed, recompute PPV at the pooled test prevalence: PPV = Se·π / (Se·π + (1 − Sp)(1 − π)).

Paper headline numbers (published version; quote exactly)

• 7 of 13 sensitivity CIs excluded zero (3 sex, 4 age); none of the 3 race/ethnicity CIs did.
• Male − Female: CDC Diabetes −0.033 [−0.048, −0.019]; Diabetes-130 −0.049 [−0.084, −0.011]; BRFSS 2022 −0.022 [−0.037, −0.008].
• Youngest − oldest: CDC −0.207 [−0.241, −0.170]; BRFSS −0.379; NHANES −0.311; CCHS −0.474; Diabetes-130 +0.104 [−0.070, +0.232] (the only positive).
• Race/ethnicity: Diabetes-130 AfricanAmerican − Caucasian −0.040 [−0.090, +0.006]; all three race intervals include zero.
• PPV standardization: BRFSS M−F +0.101 → +0.016 (both intervals exclude zero); CCHS M−F +0.053 → −0.005 (the standardized interval spans zero, so the reversal is of the point estimate only); BRFSS Black−White −0.051 → −0.009.

REQUIRED OUTPUT FILES (MANDATORY)

All in {{results_dir}}:

File	Description
result_card.html	Primary output. Self-contained, shareable one-page result card
results.json	All computed numbers (written by experiment.py)
card_data.json	Numbers/figures for the card (written by experiment.py)
narrative.json	Your interpretation of this run (Step 4)
fig_*.png	Figures (written by experiment.py)
summary.md	5–10 line plain-text summary of the run
validation_report.json	Evals v2 report: steps, code checks, checklist (see Write Validation Report)

Parameters

Parameter	Template Variable	Default	Description
Dataset	{{dataset}}	diabetes130	diabetes130 (UCI Diabetes 130-US Hospitals, 30-day readmission) or cdc_diabetes (UCI CDC Diabetes Health Indicators)
Attribute	{{attribute}}	age	sex, age, or race (race: diabetes130 only)
Age bands	{{age_bands}}	—	Age comparison bands. diabetes130: 10-29 vs 60+ (paper, default), <50 vs 70+, 10-29 vs 80+. cdc_diabetes: 18-44 vs 65+ (default), 18-24 vs 80+, 18-34 vs 70+
Group A	{{group_a}}	—	Override first group for sex/race (e.g. Asian, Hispanic, Female). Blank = paper default
Group B	{{group_b}}	—	Override reference group (e.g. Caucasian, Male). Blank = paper default
Threshold	{{threshold}}	train_prevalence	train_prevalence (paper), top20 (flag top 20% risk), or a number like 0.3
Target prevalence	{{target_prevalence}}	pooled	Common prevalence for PPV standardization: pooled (paper) or a number like 0.05
Model	{{model}}	auto	auto (paper's 6-family CV selection) or one of logistic, l1_logistic, shallow_tree, random_forest, gradient_boosting, soft_voting
Multiverse	{{multiverse}}	true	Re-run the estimate across defensible analytic choices (true/false)
Your confusion matrices	{{byo_confusion}}	—	Skip the datasets and audit your own model: GroupA:TP,FP,TN,FN; GroupB:TP,FP,TN,FN (e.g. Women:80,40,800,30; Men:120,90,700,25)

Dependencies

Everything needed (numpy, pandas, scikit-learn, matplotlib) is preinstalled in the python312-uv sandbox. Verify with python -c "import numpy, pandas, sklearn, matplotlib"; only if that fails run uv pip install --system numpy pandas scikit-learn matplotlib. Data is downloaded from the UCI Machine Learning Repository at run time (~3–13 MB).

Step 1: Set up

Write /app/work/params.json from the parameters. Substitute values exactly; use "" for blanks; n_boot, max_rows, cv_rows, multiverse_rows are fixed budget settings — do not change them:

If a value still looks like an unsubstituted placeholder (starts with {{), use the default from the table. Convert multiverse to a JSON boolean.

Step 2: Get the code (extract it, don't retype it)

The tested implementation of the method is embedded in the Appendix at the end of this runbook. Do not rewrite, "improve", or re-derive it. Always run the command below first, exactly as written. It copies the files out of this runbook's text (which the sandbox exposes to running processes) in about a second, and it is far faster and more reliable than retyping several hundred lines. Do not skip it on the assumption that it won't work: run it and read its output.

If it exits non-zero or prints MISSING, write each missing file from the Appendix exactly as given, one heredoc per file (quoted delimiter, e.g. cat > experiment.py <<'PYEOF'). If a later step fails, fix only environment problems (missing package, network retry), never the methodology.

Step 3: Run the experiment

Expected: ~30–90 s. It prints progress and ends with a done: line. If the download fails, retry once. If it exits with an input error (e.g. unknown group name, race on cdc_diabetes), fall back to the paper default for that parameter, rerun once, and record the substitution in summary.md and in a caveat.

The sandbox has 4 CPUs even though nproc may report more; the exports keep numpy/BLAS and process pools from oversubscribing them.

Step 4: Interpret: write narrative.json

Read {{results_dir}}/results.json and card_data.json, and look at every fig_*.png. Then write {{results_dir}}/narrative.json:

Rules, written for a conference audience that has not read the paper:

• headline: one sentence (≤ 30 words) stating the most surprising finding of this run, with 1–2 numbers copied from results.json (rounded as on the card). Not a description of the method. Pick the most counter-intuitive of: the multiverse splitting into both signs (multiverse_summary), the PPV gap flipping sign after standardization, the gap vanishing at another threshold, or a gap that is robust to everything.
• Keep it tight: each findings paragraph ≤ 70 words; bold the one number that matters in each.
• idea: exactly 3 bullets explaining the paper's contribution in plain language (use the Objective above).
• findings: 2–4 short paragraphs. Walk through the four questions in order: is the sensitivity gap distinguishable from zero; how much of the PPV gap is prevalence (did it shrink or flip?); does the gap depend on the threshold (quote the range from se_gap_range_over_thresholds); and, if present, how many multiverse specifications change the sign. Compare with the paper's number for the same dataset/attribute when one exists, and say plainly when your run disagrees. With default inputs the run follows the paper's published specification; differences come from model hyperparameters, the variable encodings, or inputs the participant changed (name which).
• Never call a model or subgroup result "fair", "unfair", "biased" or "discriminatory"; the paper explicitly declines that verdict. A CI including zero does not establish equivalence.
• try_next: 2–3 follow-up runs that would produce a contrasting result, each with exact parameter values using only these keys: dataset, attribute, age_bands, group_a, group_b, threshold, target_prevalence, model, multiverse, byo_confusion. Example: if the gap was robust, suggest the comparison the multiverse found most fragile; if it flipped, suggest a threshold where it disappears.
• caveats: 2–4 bullets (unweighted extracts; reconstructed outcome/feature coding; not a fairness verdict; any parameter substitutions you made).
• In BYO mode (mode == "byo"), there is no paper comparison or threshold sweep; focus on CI and prevalence.

Step 5: Render the card and summary

Write {{results_dir}}/summary.md: dataset/comparison, the sensitivity gap with CI, raw → standardized PPV gap, threshold range, multiverse sign counts, and the headline.

Writing style for narrative.json

These rules apply to every field and override any longer limits stated earlier in this runbook. Let the paper and the numbers speak; the card is not the place for commentary.

• Length: headline one sentence; idea 1–3 bullets of at most 30 words; findings 1–2 paragraphs of at most 60 words each; caveats 1–2 bullets of at most 20 words. Cut anything the tiles, figures or tables already show.
• Plain statements of what was measured. Write "Sensitivity gap −0.284 [−0.354, −0.218]; the paper reports +0.088", not an interpretation of what it means for the field.
• Avoid: em-dash asides; "not X, but Y" framings; colon-then-reveal sentences; rhetorical questions; words such as crucially, notably, strikingly, remarkably, collapses, illustrating, underscores, highlights, delve, powerful, key insight, worth noting; exclamation marks; restating the method before the result.
• Use the paper's own terms for its concepts. Quote the paper's numbers exactly.

Code Checks

Jetty runs these checks itself in the sandbox after the agent finishes, so they cannot be skipped or misreported. validate.py (see Write Validation Report) runs the same checks first so you can fix problems before you stop.

outputs-exist — Result card, numbers, narrative and summary were written

results-valid — Experiment numbers are valid JSON and the experiment stayed within its time budget

narrative-complete — The narrative is complete and within the length limits

try-next-valid — Every try-next suggestion uses only this runbook's parameters

headline-grounded — Every number in the headline appears in the run's results

card-integrity — The card embeds its figures, shows the headline, and contains no NaN

Checklist

• The headline states this run's finding with its numbers, not a description of the method
• Paper numbers are quoted exactly and any disagreement with the paper is stated plainly
• Caveats say what is synthetic, reconstructed or an extension of the paper
• Each try-next suggestion would plausibly give a contrasting result

Write Validation Report

Judge the four Checklist items honestly against your outputs and write them to /app/work/checklist.json as a list, using the checklist ids Jetty gives you (one entry per item):

Then write the report and self-check in one command:

validate.py writes {{results_dir}}/validation_report.json in the v2 shape Jetty reads (version: 2, one step entry per runbook step, one code_check entry per check above, one checklist entry per item). Jetty replaces the code_check entries with its own independent run of the same checks. Do not write the report by hand.

Iteration (up to 2 rounds)

If validate.py prints any FAIL, fix the named problem (usually a headline number that does not match results.json, or an unknown try_next key), re-render the card (Step 5) and rerun the command above. Never rerun the experiment just to change the narrative.

Appendix: code

These files are extracted verbatim in Step 2.

/app/work/experiment.py

/app/work/render_card.py

/app/work/validate.py