version	1.0.0
evaluation	programmatic
evals_version	2
agent	claude-code
model	anthropic/claude-haiku-5.5
model_provider	openrouter
snapshot	python312-uv
primary_outputs	result_card.html

Hidden Shift Finder: Missed by the Population Test, Found in a Profile (SMASH 2026 · Paper #9)

Objective

This runbook is an interactive companion to the SMASH 2026 paper "Diagnosing Low-Confidence Patient Profiles via Localized Harmful Covariate Shift Detection" (Olivier Lefebvre, Hakima Laribi, Dan Poenaru, Martin Vallières). Read the paper: https://doi.org/10.48550/arXiv.2610.19178

The paper's claim: population-level reliability checks can miss harm that is concentrated in some patients. It combines MED3pa (per-patient confidence + an interpretable tree that groups patients into low-confidence profiles) with Detectron (a learned test for harmful covariate shift, HCS) run inside each profile. In MIMIC-IV, removing patients under 50 from training was not detected at the population level for any task, yet was localized to specific degraded profiles (30-day readmission: a profile covering 17.68% of patients, AUC 0.80 → 0.64). Sex and race shifts were detected almost everywhere, so they gave little guidance on who is affected.

MIMIC-IV cannot be used here, so the participant picks a synthetic MIMIC-like cohort with known mechanisms (default) or the public UCI Diabetes 130-US hospitals data, chooses which subgroup to remove from training, and the run executes the whole pipeline, then repeats it for all three shift types (the paper's localized-vs-broad contrast). Output: a shareable result card.

Target wall-clock: under 4 minutes. The experiment itself takes about 15–30 s.

The method as in the paper (§2), and what is simplified

• Patient-level 50/50 split into training (P) and testing (Q). For each shift, the subgroup is removed from the training half only: age<50, sex (female patients removed, sex ≠ male), or race (non-White removed). Q is identical across conditions. BaseModel: MLP (here sklearn, 32 hidden units).
• Q is split in half: MED3pa learns on the first half and profiles are evaluated on the second. Confidence = regressor predicting per-patient correctness 1 − |y − p̂|; a depth-limited regression tree on those predictions gives interpretable rule-based profiles (leaves). Simplified: MED3pa's IPC/APC models are reduced to one gradient-boosting regressor + one regression tree; profiles are the leaves.
• Detectron per profile S (paper §2): P_S is split in half, P_S^r held out as reference, the rest joins all out-of-profile training-distribution cases in P_S^t. An ensemble of constrained disagreement classifiers (CDCs), each initialised from the BaseModel, is trained to agree with the BaseModel on P_S^t and to disagree on Q_S and P_S^r; cases a CDC disagrees on are rejected and the next CDC trains on the rest. HCS if significantly more Q_S than P_S^r cases are rejected (one-sided test, α = 0.05). Only profiles with ≥ 5% of patients are tested. Simplified: CDCs are fine-tuned copies of the sklearn MLP (3 epochs; each disagreement case weighted like 4 agreement cases); the test is a one-sided Fisher exact test on rejection counts (exact under no shift, because Q_S and P_S^r go through identical treatment), repeated with fresh subsamples and CDCs, HCS if the median p < α. Training-distribution performance is measured on held-out training-distribution patients (not the BaseModel's own fitting data).

Paper headline numbers (quote exactly; MIMIC-IV)

• Age shift: not detected at the population level for any of the three tasks; localized profiles detected.
• 30-day readmission, age shift: profile covering 17.68% (hemoglobin ≤ 10.91 g/dL, length of stay ≤ 5 days, no specified malignant neoplasm); mean age 73.3 → 67.8; AUC 0.80 → 0.64, balanced accuracy 0.72 → 0.58, sensitivity 0.67 → 0.42, specificity 0.77 → 0.75.
• 48-h mortality, age shift: one profile covering 8.76% (GCS-Motor ≤ 5 and age ≤ 66). 48-h live discharge: GCS-Verbal ≤ 3 profile, sensitivity decreased to zero; no HCS in profiles covering 39.2%.
• Sex shift: detected in all profiles (live discharge, mortality), AUC and sensitivity down ≈ 0.05–0.07; readmission: all but two profiles (≈ 23.8% of patients), which still collapsed (AUC and sensitivity ≈ −0.25).
• Race shift: HCS detected in every profile across all three tasks.

REQUIRED OUTPUT FILES (MANDATORY)

All in {{results_dir}}:

File	Description
result_card.html	Primary output. Self-contained, shareable one-page result card
results.json	All computed numbers (written by experiment.py)
card_data.json	Numbers/figures for the card (written by experiment.py)
narrative.json	Your interpretation of this run (Step 4)
fig_*.png	Figures (written by experiment.py)
summary.md	5–10 line plain-text summary
validation_report.json	Evals v2 report: steps, code checks, checklist (see Write Validation Report)

Parameters

Parameter	Template Variable	Default	Description
Dataset	{{dataset}}	synthetic	synthetic (100,000-patient MIMIC-like readmission cohort with known mechanisms) or diabetes130 (UCI Diabetes 130-US hospitals, real, ~3 MB download)
Removed subgroup	{{removed_subgroup}}	age<50	Subgroup removed from training only: age<50, sex (female removed), race (non-White removed)
Outcome	{{outcome}}	readmit_any	diabetes130 only: readmit_any or readmit_30 (30-day)
Tree depth	{{tree_depth}}	3	Depth of MED3pa's profile tree (2–4); deeper = more, smaller profiles
CDCs	{{n_cdc}}	5	Constrained disagreement classifiers per Detectron test (2–10)
Detectron sample	{{detectron_n}}	500	Max deployment (and reference) cases per Detectron test (50–1000)
Detectron repeats	{{detectron_repeats}}	5	Independent repeats per test; HCS if median p < α (1–9)
Min profile share	{{min_profile_share}}	0.05	Smallest profile tested (paper: 5% of the population)
α	{{alpha}}	0.05	False-positive rate for declaring HCS (paper: 0.05)
Seed	{{seed}}	0	Seed for the split, models and (synthetic) cohort

Dependencies

numpy, pandas, scikit-learn, scipy, matplotlib (preinstalled in python312-uv; scipy ships with scikit-learn). diabetes130 downloads ~3 MB from the UCI repository.

Step 1: Set up

Write /app/work/params.json (numbers as JSON numbers; if a value still looks like an unsubstituted {{…}} placeholder, use the default from the table; n_patients is a fixed budget setting, do not change it):

Validate before running: dataset ∈ {synthetic, diabetes130}; removed_subgroup ∈ {age<50, sex, race}; outcome ∈ {readmit_any, readmit_30}. Clamp tree_depth to [2, 4], n_cdc to [2, 10], detectron_n to [50, 1000], detectron_repeats to [1, 9], min_profile_share to [0.02, 0.2], alpha to [0.001, 0.2]. Record every correction as a caveat.

Step 2: Get the code (extract it, don't retype it)

The tested implementation of the method is embedded in the Appendix at the end of this runbook. Do not rewrite, "improve", or re-derive it. Always run the command below first, exactly as written. It copies the files out of this runbook's text (which the sandbox exposes to running processes) in about a second, and it is far faster and more reliable than retyping several hundred lines. Do not skip it on the assumption that it won't work: run it and read its output:

If it exits non-zero or prints MISSING, write each missing file from the Appendix exactly as given, one heredoc per file (quoted delimiter, e.g. cat > experiment.py <<'PYEOF'). If a later step fails, fix only environment problems (missing package, network retry), never the methodology.

Step 3: Run the experiment

Expected: 15–40 s. Prints one line per shift type and ends with a done: line. If the diabetes130 download fails, retry once; if it fails again, rerun with "dataset": "synthetic" and record the substitution as a caveat.

The sandbox has 4 CPUs even though nproc may report more; the exports keep numpy/BLAS and process pools from oversubscribing them.

Step 4: Interpret: write narrative.json

Read {{results_dir}}/results.json and card_data.json, and look at every fig_*.png. Then write {{results_dir}}/narrative.json with keys headline, idea (exactly 3), findings (2–4), try_next (2–3), caveats (2–4):

• headline (≤ 30 words): the localization result of this run with 1–2 numbers from results.json. If localized is true: the population-level test missed the shift (its median p_value) but a profile was flagged, with its AUC change (headline_profile.train_dist.auc → headline_profile.test.auc) and share. If the population test also fired, or nothing was flagged, say that plainly.
• idea: plain language: a model can be fine on average and unreliable for some patients; MED3pa finds interpretable low-confidence patient groups; Detectron, run inside each group, tests whether a harmful shift between training and deployment patients plausibly explains that group's problem.
• findings: (1) population vs profile verdicts (population.detectron, each profile's status and detectron); (2) the flagged profile: its rule, share, share of the removed subgroup (subgroup_share_test), mean age training → deployment, AUC / sensitivity / specificity vs the paper's 0.80 → 0.64 etc.; (3) the three-shift contrast (contrast): age localized vs sex/race broad, as in the paper; (4) the paper's caution: HCS without degradation, or degradation without HCS, can both happen; check the profiles in fig_profiles.png for such cases.
• Never call a group "biased" or a model "unfair"; describe reliability and shift.
• try_next: 2–3 contrasting runs with exact values, using only the keys dataset, removed_subgroup, outcome, tree_depth, n_cdc, detectron_n, detectron_repeats, min_profile_share, alpha, seed. Good contrasts: the real data (dataset diabetes130), a broad shift (removed_subgroup sex), a smaller Detectron sample (detectron_n 100, where power drops), a deeper tree (tree_depth 4).
• caveats: synthetic cohort with planted mechanisms (or Diabetes-130 rather than MIMIC-IV); MED3pa and Detectron are faithful simplifications (sklearn MLP CDCs, Fisher test, median of repeats); results vary with seed, so a borderline p-value deserves a second seed; HCS is a diagnostic signal, not proof that the shift caused the error.

Step 5: Render the card and summary

Write {{results_dir}}/summary.md (5–10 lines: dataset and removed subgroup, population verdict, flagged profiles with rules and AUC change, three-shift contrast, headline).

Writing style for narrative.json

These rules apply to every field and override any longer limits stated earlier in this runbook. Let the paper and the numbers speak; the card is not the place for commentary.

• Length: headline one sentence; idea 1–3 bullets of at most 30 words; findings 1–2 paragraphs of at most 60 words each; caveats 1–2 bullets of at most 20 words. Cut anything the tiles, figures or tables already show.
• Plain statements of what was measured. Write "Sensitivity gap −0.284 [−0.354, −0.218]; the paper reports +0.088", not an interpretation of what it means for the field.
• Avoid: em-dash asides; "not X, but Y" framings; colon-then-reveal sentences; rhetorical questions; words such as crucially, notably, strikingly, remarkably, collapses, illustrating, underscores, highlights, delve, powerful, key insight, worth noting; exclamation marks; restating the method before the result.
• Use the paper's own terms for its concepts. Quote the paper's numbers exactly.

Code Checks

Jetty runs these checks itself in the sandbox after the agent finishes, so they cannot be skipped or misreported. validate.py (see Write Validation Report) runs the same checks first so you can fix problems before you stop.

outputs-exist — Result card, numbers, narrative and summary were written

results-valid — Experiment numbers are valid JSON and the experiment stayed within its time budget

narrative-complete — The narrative is complete and within the length limits

try-next-valid — Every try-next suggestion uses only this runbook's parameters

headline-grounded — Every number in the headline appears in the run's results

card-integrity — The card embeds its figures, shows the headline, and contains no NaN

Checklist

• The headline states this run's finding with its numbers, not a description of the method
• Paper numbers are quoted exactly and any disagreement with the paper is stated plainly
• Caveats say what is synthetic, reconstructed or an extension of the paper
• Each try-next suggestion would plausibly give a contrasting result

Write Validation Report

Judge the four Checklist items honestly against your outputs and write them to /app/work/checklist.json as a list, using the checklist ids Jetty gives you (one entry per item):

Then write the report and self-check in one command:

validate.py writes {{results_dir}}/validation_report.json in the v2 shape Jetty reads (version: 2, one step entry per runbook step, one code_check entry per check above, one checklist entry per item). Jetty replaces the code_check entries with its own independent run of the same checks. Do not write the report by hand.

Iteration (up to 2 rounds)

If validate.py prints any FAIL, fix the named problem (usually a headline number that does not match results.json, or an unknown try_next key), re-render the card (Step 5) and rerun the command above. Never rerun the experiment just to change the narrative.

Appendix: code

These files are extracted verbatim in Step 2.

/app/work/experiment.py

/app/work/render_card.py

/app/work/validate.py