SMASH 2026 · Paper #20 · Result card

Towards Robust, Performance-Based Contamination Detection for LLMs

Allenna Tang, Jackie Cheung · McGill University · Mila · Read the paper

The evasion drops the confidence detectors from 1.00 to 0.52 AUROC without changing an answer. The IRT residual detector stays at 0.73.

population 102 modelsitems 1,200 members / 1,200 unseen / 3,600 calibrationcontamination strength 0.97model ability θ 0.3memorization boost 1.0seed 7

Best confidence detector

1.00 → 0.52

AUROC before → after the evasion

IRT-residual detector

0.73 → 0.73

unchanged by construction (p = 0.002)

Benchmark answers changed by evasion

of 2,400 audited items

Cost to beat IRT too

−13.1 pts

adversary must give back 70% of its contamination gains

From the abstract

Our evasion is an argmax-preserving logit transform that makes signals on contaminated items resemble those on unseen items, while leaving every benchmark answer unchanged. We then introduce a detector based on Item Response Theory (IRT) residuals, which calibrates item difficulty on a population of 102 LLMs and flags items where a model outperforms its estimated ability. Because it reads only binary responses, its AUROC is exactly unchanged under the evasion.

Evidence

The evasion: confidence detectors collapse, IRT does not

Paper vs. your run

Metric	Paper	Your run
Min-K%++ AUROC (before → after)	0.99 → 0.48	0.96 → 0.50
Min-K% AUROC (before → after)	0.93 → 0.54	0.97 → 0.51
LOSS AUROC (before → after)	0.99 → 0.54	1.00 → 0.52
ZLIB AUROC (before → after)	0.99 → 0.45	1.00 → 0.52
IRT-residual AUROC (before → after)	0.68 → 0.68 (Δ 0.00)	0.73 → 0.73
Control (uncontaminated) IRT AUROC	0.5016, p = 0.495	0.484, p = 0.890
Easy-item blind spot (IRT AUROC by difficulty)	on very easy items r ≈ 0: contamination undetectable	hard 0.83 / medium 0.66 / easy 0.60
Member accuracy (contamination)	0.5996 → 0.9908	0.615 → 0.990	synthetic

Try next

Light contamination — With contamination=0.3 (only 30% of wrong member answers become right), the IRT signal should weaken. Tests whether the detector still achieves significance when contamination is modest.
```
{"contamination": 0.3, "model_ability": 0.3, "memorization_boost": 1.0, "n_member_items": 1200, "n_calibration_items": 3600, "n_population": 102, "seed": 7}
```
Very strong model (most items easy) — With model_ability=2.0 most items are easy for this model, so contamination residuals are small everywhere — the easy-item blind spot dominates and IRT AUROC should fall toward 0.5 even with strong contamination.
```
{"contamination": 0.97, "model_ability": 2.0, "memorization_boost": 1.0, "n_member_items": 1200, "n_calibration_items": 3600, "n_population": 102, "seed": 7}
```
Tiny calibration population — With n_population=20 (vs. paper's 102), item-parameter estimates are noisier, ability estimation is less reliable, and the IRT detector should degrade — showing how sensitive detection is to the breadth of the calibration pool.
```
{"contamination": 0.97, "model_ability": 0.3, "memorization_boost": 1.0, "n_member_items": 1200, "n_calibration_items": 3600, "n_population": 20, "seed": 7}
```

Read this before sharing

Items, models and token probabilities are simulated.
The sandbagging analysis is an extension of the paper.