SMASH 2026 · Paper #18 · Result card
From Extraction Accuracy to Clinical Usefulness: A Human-Centered Evaluation Framework for Source-Traceable Patient Summaries
An LLM judge scored a summary 9/10 while the contradiction gate blocked it: two conflicting HbA1c values (7.2% and 8.1%) had been reduced to one.
summarizer anthropic/claude-haiku-4.5 · brief promptjudge anthropic/claude-haiku-4.5noise conflicting labs, copied-forward note, document ≠ event date, discontinued med, negated finding, resolved diagnosisvariants 9LLM calls 18
LLM-judge score
6/10
anthropic/claude-haiku-4.5 · holistic quality, as many pipelines report it
Release gates
FAILED 2 of 8
conflicting HbA1c silently resolved to one value; no intended-use / verify-against-source statement
Gold claims covered
100%
13 claims · 0% not lexically supported by their cited segment · severity-weighted errors 2
Judged ≥ 7/10 yet a clinical gate failed
1 of 9
e.g. brief prompt on the pack with 3 noise types (up to conflicting labs): judge 9/10, failed contradiction · 4 counting the intended-use gate
From the abstract
A fact can be extracted correctly and still mislead if it is presented without temporal context, detached from its source, merged with conflicting information, or rewritten in a way that changes perceived urgency, certainty, or relevance. The framework combines deterministic checks, review of clinical meaning, and task-based evaluation of how users interpret and verify summaries …
Evidence
Paper vs. your run
| Metric | Paper | Your run | |
|---|---|---|---|
| Release gates vs averaged scores | “A gain in readability should not offset broken evidence links, a known status inversion, or loss of negation.” | brief: judge 6/10, gates FAILED (Conflict surfaced, both sources cited, Intended-use language) | framework claim; no paper numbers |
| Model-as-judge | should not serve as the sole evidence of clinical acceptability because judge models may share blind spots with the system being evaluated | judge (≥7) and gates agreed on 56% of 9 summaries | same model family judges itself |
| Missing sources | “If a claim lacks an accessible source, the presence of other well-supported claims should not compensate for it.” | citation gate passed | |
| Severity | “Errors should be stratified by likely consequence rather than counted equally.” | severity-weighted errors 2 (gold severities 1–3) | |
| Controlled noise | duplicate and copied-forward statements, missing fields, conflicting sources, resolved-versus-active ambiguity, medication changes, and differences between document date and clinical-event date | 6 noise types planted; ladder of 7 packs | Synthea-style pack written for this runbook |
Try next
- Careful prompt on the full pack — The careful prompt is the only one that passed all gates; running it with compare_styles off confirms the gate pass and shows the judge score without style contamination.
{"summarizer_style": "careful", "noise_types": "all", "compare_styles": false, "degradation_ladder": false} - Isolate the conflicting-labs noise type — The contradiction gate fired at noise step 3 while the judge scored 9/10 — the sharpest divergence. Running with only conflicting_labs active isolates that gate failure cleanly.
{"summarizer_style": "brief", "noise_types": "conflicting_labs", "compare_styles": true, "degradation_ladder": false} - Large summarizer on all noise — Upgrading to Claude Sonnet tests whether a more capable model handles the contradiction and intended-use requirements without an explicit careful prompt.
{"summarizer_style": "brief", "summarizer_size": "large", "noise_types": "all", "compare_styles": false, "degradation_ladder": false}
Read this before sharing
- The patient pack is synthetic, written for this runbook.
- Gates are simplified lexical checks.