version	1.0.0
evaluation	programmatic
evals_version	2
agent	claude-code
model	anthropic/claude-haiku-5.5
model_provider	openrouter
snapshot	python312-uv
primary_outputs	result_card.html

Private Claim Check: Score a Health Claim the Server Never Sees (SMASH 2026 · Paper #13)

Objective

This runbook is an interactive companion to the SMASH 2026 paper "Privacy-Preserving Text Inference for Health-Misinformation Classification and Semantic Retrieval" (Jasleen Kaur, Chloe Cachet, Koray Karabina, Jean Belo Klamti, Philippe Lamontagne). Read the paper: https://arxiv.org/abs/2610.20796

The paper's claims:

1. A logistic-regression misinformation classifier on fused text features (BioBERT + PCA-256 MPNet + SVD-300 TF-IDF, 1,324-d) can be evaluated on CKKS-encrypted feature vectors: the server computes z = x̃ᵀw + b on the ciphertext, the client decrypts and applies the sigmoid. Plaintext and ciphertext inference give identical accuracy and F1 on CoAID.
2. Encrypted semantic retrieval is practical once similarity scores are SIMD-packed into one ciphertext rather than returned one ciphertext per database item.

The participant brings their own health claims. The run trains the paper's classifier recipe on the public CoAID dataset, scores the participant's claims in plaintext and under encryption, checks the whole test split under encryption, benchmarks latency/ciphertext size at the paper's parameters, and runs naive vs packed encrypted retrieval of the most similar CoAID items. Output: a shareable result card.

Target wall-clock: under 4 minutes. The experiment itself takes ~15–30 s (4.3 MB CoAID + 30 MB embedder).

The method, as implemented

• Data. CoAID news (GitHub cuilimeng/CoAID, releases 05-01/07-01/09-01/11-01-2020, NewsFake/NewsReal): text = title + news title + first 2,000 characters of content (text_field = title+content) or title only; exact duplicates removed; stratified 80/20 split (seed 42) → test 1,080 items (183 fake), close to the paper's 1,081 (184).
• Client features. TF-IDF (1–2-grams) → truncated SVD (svd_dim, paper 300) and/or a 256-d static sentence embedding (model2vec potion-base-8M, stand-in for the paper's PCA-256 MPNet). Standardized on the client. Substitution: the paper's 768-d BioBERT block is not reproduced (no torch/GPU), so the hybrid is 556-d.
• Model. L2 logistic regression (C). Ablation in plaintext over TF-IDF / embedding / hybrid.
• Encrypted inference. TenSEAL CKKS, scale 2^40, moduli [60, 40, 60] (N = 8,192) or [60, 40, 40, 60] (N = 16,384 / 32,768; the paper uses N = 32,768). Client encrypts x; server returns Enc(x)·w + b; client decrypts z, computes p = σ(z). Run on the whole test split (capped at 400 items when N = 32,768) and every claim.
• Encrypted retrieval. Database = unique CoAID titles embedded (256-d, unit norm). Naive: one encrypted dot product (one ciphertext) per item, timed on 200 items and extrapolated; packed: one Enc(q)·DBᵀ per N/2-item shard. ISCC-SCT fingerprints, Ridge late fusion and STS-B are not reproduced.

Paper headline numbers (quote exactly)

• Full hybrid: accuracy 0.971, macro-F1 0.950 (F1-fake 0.918, F1-real 0.983); TF-IDF alone 0.942 / 0.904; MPNet alone 0.938 / 0.899; BioBERT alone 0.944 / 0.905. Test set of 1,081 samples, 184 fake and 897 real.
• "Plaintext and ciphertext-based inference produced identical accuracy and F1 scores for every feature configuration."
• "The hybrid model required approximately 122.4 ms and 2.133 MB of communication per query."
• Retrieval over 7,128 items: naive 221.049 s → SIMD-packed 5.186 s (42.6×); communication 6.11 GB → 2.13 MB.
• CKKS: N = 32,768, coefficient moduli [60, 40, 40, 60], scale 2^40, Apple M4 MacBook Air.

REQUIRED OUTPUT FILES (MANDATORY)

All in {{results_dir}}:

File	Description
result_card.html	Primary output. Self-contained, shareable one-page result card
results.json	All computed numbers (written by experiment.py)
card_data.json	Numbers/figures for the card (written by experiment.py)
narrative.json	Your interpretation of this run (Step 4)
fig_*.png	Figures (written by experiment.py)
summary.md	5–10 line plain-text summary
validation_report.json	Evals v2 report: steps, code checks, checklist (see Write Validation Report)

Parameters

Parameter	Template Variable	Default	Description
Your claims	{{claims}}	—	Up to 10 health claims separated by | (blank = 6 built-in examples, true and false). Scored under encryption
Features	{{features}}	hybrid	hybrid (TF-IDF→SVD + embedding), tfidf, or embed
Training text	{{text_field}}	title+content	title+content (paper-like) or title (short, closer to what people paste)
SVD dimensions	{{svd_dim}}	300	TF-IDF truncated-SVD size (paper: 300)
CKKS ring size	{{poly_modulus_degree}}	8192	8192, 16384 or 32768 (paper). Used for the test-split and retrieval runs; the latency benchmark always covers all three
Retrieval database size	{{retrieval_db_size}}	0	Number of CoAID titles to search (0 = all ~5,200)
Regularization C	{{C}}	1.0	Inverse L2 strength of the logistic regression
Embedder	{{embedder}}	minishlab/potion-base-8M	model2vec static embedder (30 MB)
Seed	{{seed}}	42	Train/test split seed

Dependencies

numpy, pandas, scikit-learn, matplotlib (preinstalled in python312-uv), plus tenseal (0.3.18) and model2vec installed in Step 1 (~10–30 s). Downloads: 8 CoAID CSVs from raw.githubusercontent.com (4.3 MB) and the embedder from Hugging Face (30 MB).

Step 1: Set up

Write /app/work/params.json. Strings for text values, JSON numbers for the rest; if a value still looks like an unsubstituted {{…}} placeholder use the table default (claims → ""). Validate: features ∈ {hybrid, tfidf, embed}; text_field ∈ {title+content, title}; poly_modulus_degree ∈ {8192, 16384, 32768}; clamp svd_dim to [20, 600], retrieval_db_size to 0 or [200, 6000], C to [0.001, 100]. Keep at most 10 claims, each ≤ 300 characters; escape quotes for JSON. Record any change as a caveat. n_encrypted_eval and naive_items are fixed budget settings.

Step 2: Get the code (extract it, don't retype it)

The tested implementation of the method is embedded in the Appendix at the end of this runbook. Do not rewrite, "improve", or re-derive it. Always run the command below first, exactly as written. It copies the files out of this runbook's text (which the sandbox exposes to running processes) in about a second, and it is far faster and more reliable than retyping several hundred lines. Do not skip it on the assumption that it won't work: run it and read its output.

If it exits non-zero or prints MISSING, write each missing file from the Appendix exactly as given, one heredoc per file (quoted delimiter, e.g. cat > experiment.py <<'PYEOF'). If a later step fails, fix only environment problems (missing package, network retry), never the methodology.

Step 3: Run the experiment

Expected: 15–30 s at N = 8,192 (≈ 60–90 s at N = 32,768). Ends with a done: line (accuracy/macro-F1, encrypted labels identical, claims flagged, packed-retrieval speed-up). If a download fails, retry once.

The sandbox has 4 CPUs even though nproc may report more; the exports keep numpy/BLAS and process pools from oversubscribing them.

Step 4: Interpret: write narrative.json

Read {{results_dir}}/results.json and card_data.json, and look at every fig_*.png. Write {{results_dir}}/narrative.json with keys headline, idea (exactly 3), findings (2–4), try_next (2–3), caveats (2–4). Each findings paragraph ≤ 70 words; bold the one number that matters.

• headline (≤ 30 words): a finding with 1–2 numbers copied from results.json, e.g. encrypted.labels_identical of encrypted.n, retrieval.speedup, or a striking claim result. Write numbers without thousands separators in the headline (1080, not 1,080) and avoid scientific notation there (the validator parses digits).
• idea: plain-language version of the two claims above (why the text is sensitive; what the server computes; why packing matters).
• findings: (1) accuracy/macro-F1 of the chosen features vs the paper's row (ablation; TF-IDF row is the like-for-like comparison) and the test split vs 1,081/184; (2) encryption is lossless: labels_identical, max_abs_logit_diff, and each claim's label_enc, p_fake_enc, abs_diff; name which of the participant's claims were flagged; mention the nearest CoAID items for one claim (retrieved, with their CoAID labels) if telling; (3) cost: N = 32,768 row of latency_bench vs 122.4 ms / 2.133 MB; retrieval naive vs packed vs 221.049 s → 5.186 s and same_top5.
• Never present the output as a fact-check. “Fake” means “resembles the CoAID fake class”; CoAID labels partly encode the source and writing style (see top_terms). If a true claim is flagged or a false one passes, say so plainly.
• try_next: 2–3 contrasting runs with exact values, using only the keys claims, features, text_field, svd_dim, poly_modulus_degree, retrieval_db_size, C, embedder, seed. Good contrasts: text_field title with false-cure claims (style vs truth); poly_modulus_degree 32768; features tfidf; the same claim reworded as a social post.
• caveats: feature substitutions (no BioBERT, static embedder instead of MPNet, 556-d not 1,324-d); retrieval without ISCC-SCT/STS-B; honest-but-curious threat model; timings on one CPU, naive retrieval extrapolated.

Step 5: Render the card and summary

Write {{results_dir}}/summary.md (5–10 lines: settings, accuracy/macro-F1 vs paper, encrypted agreement, each claim's label, latency/size at N = 32,768, retrieval speed-up, headline).

Writing style for narrative.json

These rules apply to every field and override any longer limits stated earlier in this runbook. Let the paper and the numbers speak; the card is not the place for commentary.

• Length: headline one sentence; idea 1–3 bullets of at most 30 words; findings 1–2 paragraphs of at most 60 words each; caveats 1–2 bullets of at most 20 words. Cut anything the tiles, figures or tables already show.
• Plain statements of what was measured. Write "Sensitivity gap −0.284 [−0.354, −0.218]; the paper reports +0.088", not an interpretation of what it means for the field.
• Avoid: em-dash asides; "not X, but Y" framings; colon-then-reveal sentences; rhetorical questions; words such as crucially, notably, strikingly, remarkably, collapses, illustrating, underscores, highlights, delve, powerful, key insight, worth noting; exclamation marks; restating the method before the result.
• Use the paper's own terms for its concepts. Quote the paper's numbers exactly.

Code Checks

Jetty runs these checks itself in the sandbox after the agent finishes, so they cannot be skipped or misreported. validate.py (see Write Validation Report) runs the same checks first so you can fix problems before you stop.

outputs-exist — Result card, numbers, narrative and summary were written

results-valid — Experiment numbers are valid JSON and the experiment stayed within its time budget

narrative-complete — The narrative is complete and within the length limits

try-next-valid — Every try-next suggestion uses only this runbook's parameters

headline-grounded — Every number in the headline appears in the run's results

card-integrity — The card embeds its figures, shows the headline, and contains no NaN

Checklist

• The headline states this run's finding with its numbers, not a description of the method
• Paper numbers are quoted exactly and any disagreement with the paper is stated plainly
• Caveats say what is synthetic, reconstructed or an extension of the paper
• Each try-next suggestion would plausibly give a contrasting result

Write Validation Report

Judge the four Checklist items honestly against your outputs and write them to /app/work/checklist.json as a list, using the checklist ids Jetty gives you (one entry per item):

Then write the report and self-check in one command:

validate.py writes {{results_dir}}/validation_report.json in the v2 shape Jetty reads (version: 2, one step entry per runbook step, one code_check entry per check above, one checklist entry per item). Jetty replaces the code_check entries with its own independent run of the same checks. Do not write the report by hand.

Iteration (up to 2 rounds)

If validate.py prints any FAIL, fix the named problem (usually a headline number that does not match results.json, or an unknown try_next key), re-render the card (Step 5) and rerun the command above. Never rerun the experiment just to change the narrative.

Appendix: code

These files are extracted verbatim in Step 2.

/app/work/experiment.py

/app/work/render_card.py

/app/work/validate.py