SMASH 2026 · Paper #9 · Result card

Diagnosing Low-Confidence Patient Profiles via Localized Harmful Covariate Shift Detection

Olivier Lefebvre, Hakima Laribi, Dan Poenaru, Martin Vallières · Université de Sherbrooke · Montreal Children's Hospital · McGill University · Read the paper

The age shift was missed at population level (p = 0.19). Two of five profiles were flagged; in the largest, AUC fell from 0.69 to 0.64.

dataset syntheticoutcome 30-day readmission (synthetic MIMIC-like cohort)removed from training patients under 50tree depth 3CDCs 5Detectron sample 500min profile 5%α 0.05seed 0

Population-level Detectron

no HCS

median p = 0.192 over 5 repeats; CDCs rejected 10% of deployment vs 8% of reference cases

Profiles with a localized HCS

2 of 5

tested profiles (≥ 5% of patients), covering 47% of the population

Worst flagged profile: AUC

0.69 → 0.64

Hb ≤ 11.55 & creat ≤ 1.695 · 31% of patients · sensitivity 0.83 → 0.79

Population AUC

0.75 → 0.73

training distribution → deployment; removed subgroup alone 0.67

From the abstract

MED3pa first discovers interpretable low-confidence patient profiles, and Detectron is then applied within each profile to test whether its degradation is associated with a detectable harmful covariate shift. An injected age shift, undetectable at the population level, was nonetheless localized to specific degraded profiles …

Evidence

Can disagreement classifiers tell deployment from training cases?

Localized or broad? Three injected shifts

Paper vs. your run

Metric	Paper	Your run
Age shift detected at population level?	No, for all three tasks	no HCS (median p = 0.192)
Age shift localized to profiles	Yes: e.g. readmission profile covering 17.68%	2 of 5 profiles (47% of patients)
Sex / race shifts	Detected in all (or all but two) profiles: broad, little localizing value	sex 7/7 · race 5/7 profiles; population HCS / HCS
Degraded profile: AUC / balanced acc. / sensitivity / specificity	0.80→0.64 / 0.72→0.58 / 0.67→0.42 / 0.77→0.75 (readmission, age shift)	0.69→0.64 / 0.61→0.59 / 0.83→0.79 / 0.39→0.39	Hb ≤ 11.55 & creat ≤ 1.695
Mean age in that profile, training → deployment	73.3 → 67.8	68.9 → 61.1

Try next

Real data: UCI Diabetes 130-US hospitals — Tests whether localization holds on a real, noisier dataset without planted mechanisms (the paper used MIMIC-IV, not synthetic data).
```
{"dataset": "diabetes130", "removed_subgroup": "age<50", "outcome": "readmit_any", "seed": 0}
```
Broad shift (sex removed) as primary condition — Confirms the broad-vs-localized contrast: the sex shift fires everywhere in the contrast table, so running it as the primary condition should yield population HCS and profiles flagged across the board.
```
{"dataset": "synthetic", "removed_subgroup": "sex", "seed": 0}
```
Lower Detectron power with small sample — Reducing detectron_n to 100 lowers statistical power; borderline profiles may drop below significance, showing how sample size affects the ability to localize HCS.
```
{"dataset": "synthetic", "removed_subgroup": "age<50", "detectron_n": 100, "seed": 1}
```

Read this before sharing

A synthetic MIMIC-like cohort replaces MIMIC-IV.
Detectron is simplified; borderline profiles change with the seed.