SMASH 2026 · Paper #19 · Result card
When Engagement Is Not Enough: What Should AI Optimize in Gamified Education?
With equal weights the gamified condition wins 55% of resamples. The verdict flips once learning gets more than 51% of the weight.
weights E/L/B/P 0.25 / 0.25 / 0.25 / 0.25retention (assumed) 0.5affect–learning corr. 0.0engagement scale top 7.0n Case 1 49n Case 2 37seed 3data synthetic, calibrated to the paper
Winner under your weights
Gamified
in 55% of 4,000 bootstrap resamples (Case 1, composite +0.017)
Decision flips at
51% on L
you put 50% on learning vs engagement; 50% of your weight is on outcomes Case 1 never measured
Engagement as a proxy
43%
of the top-third knowledge gainers found by watching engagement (chance 33%; 23% of learners hit the ceiling)
Same game, two verdicts
3.68 vs 0.03
Case 2 standardized effect: engagement vs behavioural intention
From the abstract
In crossover STEAM education (n = 49), interest favored gamified instruction (dz = 0.64), whereas perceived expression acquisition favored conventional instruction (dz = -0.55). In disaster education (n = 37), engagement was high and perceived knowledge and competence increased, but intention changed little.
Evidence
Paper vs. your run
| Metric | Paper | Your run | |
|---|---|---|---|
| Case 1 d_z: Interest | 0.64 [0.35, 0.92] | 0.64 [0.34, 1.05] | synthetic, n = 49 |
| Case 1 d_z: Enjoyment of interaction | 0.61 [0.32, 0.89] | 0.61 [0.33, 0.94] | synthetic, n = 49 |
| Case 1 d_z: Overall enjoyment | 0.50 [0.21, 0.78] | 0.50 [0.22, 0.82] | synthetic, n = 49 |
| Case 1 d_z: Acquisition of new expressions | -0.55 [-0.84, -0.27] | -0.55 [-0.90, -0.28] | synthetic, n = 49 |
| Case 2 knowledge T0 → T1 (d_z) | 4.37 → 5.45 (d_z = 1.53) | 4.37 → 5.44 (d_z = 1.53) | calibrated |
| Case 2 competence T0 → T1 (d_z) | 4.07 → 5.49 (d_z = 1.99) | 4.07 → 5.48 (d_z = 1.99) | calibrated |
| Case 2 intention T0 → T1 (d_z) | 5.97 → 6.00 (d_z = 0.03) | 5.97 → 6.00 (d_z = 0.03) | calibrated |
| r(engagement, gain): knowledge / intention / competence | r ≈ .24 / .26 / .05 | 0.17 / 0.24 / -0.01 | one synthetic cohort, engagement scale 1–7 |
| Persistence (T2) | T2 is ongoing; at present, persistence cannot yet be established | assumed retention 0.5 | your assumption, not data |
Try next
- Tip past the flip: weight learning more than engagement — With the flip at L-share 51% and your current position at 50%, shifting just a little weight from E to L reverses the winner. This run makes the reversal explicit.
{"w_engagement": 0.2, "w_learning": 0.3, "w_intention": 0.25, "w_persistence": 0.25, "seed": 3} - Scale up to n = 200 — does more data firm up the verdict? — At n = 49 the gamified win is only 55% across resamples. A larger synthetic cohort shows whether the uncertainty shrinks or whether the close d_z values mean both conditions are genuinely plausible.
{"w_engagement": 0.25, "w_learning": 0.25, "w_intention": 0.25, "w_persistence": 0.25, "n_case1": 200, "seed": 3} - Raise the engagement ceiling to 10 — does the proxy improve? — 23% of Case 2 learners hit the 7-point ceiling. Using a 10-point scale removes most of that saturation; check whether the proxy capture rate rises meaningfully or stays near 43%.
{"w_engagement": 0.25, "w_learning": 0.25, "w_intention": 0.25, "w_persistence": 0.25, "engagement_ceiling": 10, "seed": 3}
Read this before sharing
- Learners are simulated from the paper's summary statistics.
- Persistence is an assumed value; the paper's follow-up data are not modelled.