SMASH 2026 · Paper #11 · Result card
MASQ: The Multi-Agent System Questionnaire
An LLM filling in MASQ for ChatDev anticipated 8 of 14 failure modes, the same count as the authors. In a small ChatDev-style chain, 3 of 5 tasks declared done failed hidden tests.
system ChatDev (paper)LLM anthropic/claude-haiku-4.5Stage 3 5 mini-MAS tracesverification promptmemory between phases dropreview rounds 2LLM calls 35
LLM screen agrees with authors
9/14
exact Low/Mid/High match with the paper's Stage 2; 13/14 within one level
Anticipated correctly (ChatDev)
8/14
LLM screen vs MAST's 30 annotated ChatDev traces; authors: 8/14. Verification (FM-3.2) rated Low (authors Low, observed 11/30)
Mini-MAS: anticipated = observed
2/14
5 traces of a ChatDev-shaped chain (prompt verification, history drop)
Declared done, hidden tests fail
3 of 5
hidden-test pass rate 88%; reviewer approved 5/5
From the abstract
We introduce the multi-agent system questionnaire (MASQ), a three-stage screening framework that connects system design choices to plausible miscoordination risks and their observable manifestations. Stage 3 guides researchers in examining MAS execution traces for evidence of anticipated and unanticipated miscoordination.
Evidence
Paper vs. your run
| Metric | Paper | Your run | |
|---|---|---|---|
| Correctly anticipated (ChatDev) | Stage 2 correctly predicts 8/14 MAST risks | LLM screen: 8/14 | scored against the paper's own Stage 3 counts |
| No or incomplete verification (FM-3.2) | Low; occurring in 11/30 traces | LLM: Low | the paper's main miss |
| Loss of conversation history (FM-1.4) | High; 2/30 (!) | LLM: Low | |
| Conversation reset (FM-2.1) | High; 1/30 (!) | LLM: High | |
| Overestimated risks | we overestimated five risk categories | LLM screen overestimated 4, underestimated 2 | (!) cells in the grid |
| Role prompts vs mechanical verification | role-prompting does not substitute for formal, mechanical verification | 3/5 mini-MAS traces declared done but failed hidden tests (prompt verification) | runbook extension |
Try next
- Mechanical test execution (mas_verification=tests) — Switching to executed assert statements should close the declared-done-but-hidden-fail gap and plausibly lower FM-3.2's anticipated risk to Low—testing whether the screen now correctly predicts that verification failure.
{"mas_verification": "tests", "mas_memory": "drop", "n_traces": 5} - Shared history across phases (mas_memory=shared) — The current drop setting drove High ratings for FM-1.4 and FM-2.1 (memory modes); shared history should rate those Low and may reduce FM-2.4 (information withholding), revealing whether the memory dimension drives the mismatch surplus.
{"mas_memory": "shared", "mas_verification": "prompt", "n_traces": 5} - Group-chat topology instead of chat chain — The groupchat preset has a shared ever-growing transcript and a manager agent—a different network topology and memory architecture—likely shifting High risks away from FM-1.4/FM-2.1 and toward FM-1.3 (step repetition) and FM-3.1 (premature termination).
{"system_preset": "groupchat", "run_stage3": false}
Read this before sharing
- Stage 2 ratings are an LLM applying the paper's rule, not the authors' labels.
- Five traces give coarse rates.