StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation
Abstract
Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.
Method
Jointly modeling game states and visual dynamics for mechanics-consistent generation.
- State & Visual Branches. A lightweight state expert complements the visual expert, adding explicit state modeling with little overhead (0.75B).
- Mixture of Transformers. Separate experts preserve modality-specific features, while bidirectional joint attention exchanges state and visual cues, allowing predicted states to guide frame generation.
- Loss Design. Flow matching loss handles high-dimensional video, while Smooth L1 loss directly supervises compact, rule-governed states.
Visualization
Each pair compares a stateless game world model with StatePlay. Both columns are synchronized.
Mechanics: a super art is valid only when the skill meter is full; after execution, the meter should reset to zero.
Mechanics: when the NPC health reaches zero, combat must terminate and a “You Win” result should be displayed.
Mechanics: when the player health reaches zero, combat must terminate and a “You Lose” result should be displayed.
Quantitative Results
Evaluation covers visual quality, action control, state alignment, and mechanics fidelity on 100 held-out five-second gameplay clips.
Table 1. Comparison with Game World Models
Bold values denote the best result. Mechanics fidelity is reported as mean ± standard deviation over three runs; “—” indicates a metric that is not applicable.
| Method | Visual Quality | Action Control (%) | State Alignment ↑Compares predicted timer, HP, and skill-meter traces with their targets using normalized error, then reports 1 − average distance. | Mechanics Fidelity (%) (n=3) | |||
|---|---|---|---|---|---|---|---|
| SSIM ↑Compares generated frames with aligned and cropped game-engine references for structural similarity, including layout, character shape, and local structure. | LPIPS ↓Compares generated and reference frames using deep visual features to measure perceptual distance. | Move-Acc ↑Tracks the generated character with SAM 2.1 and Grounding DINO, then checks whether normalized displacement matches the commanded direction. | Att-Acc ↑Uses ClipAttackNet, a six-class frame-level attack classifier, to check whether generated attacks match the input command. | Gemini-3.1-Pro ↑Evaluates each generated video directly against reference images and target mechanic labels, including correct win/lose outcomes and valid super-art execution. | GPT-5.5 ↑Evaluates 24 uniformly sampled frames from each video against the same references and target mechanic labels. | ||
| Zero-shot Evaluation | |||||||
| Matrix-Game 3.0 | 0.142 | 0.673 | — | — | — | 42.3 ± 0.9 | 42.3 ± 0.5 |
| LingBot-World | 0.183 | 0.601 | — | — | — | 44.3 ± 0.5 | 53.0 ± 0.8 |
| LingBot-World 2.0 | 0.191 | 0.641 | — | — | — | 41.3 ± 0.9 | 43.0 ± 0.8 |
| HY-World 1.5 | 0.172 | 0.537 | — | — | — | 41.0 ± 0.8 | 43.0 ± 1.4 |
| ReactiveGWM | 0.340 | 0.457 | — | — | — | 48.0 ± 0.7 | 43.3 ± 1.6 |
| State-aware Fine-tuning | |||||||
| Matrix-Game 3.0 | 0.240 | 0.478 | 40.0 | 6.67 | — | 48.7 ± 1.7 | 58.3 ± 0.9 |
| HY-World 1.5 | 0.352 | 0.521 | 40.0 | 11.7 | — | 41.7 ± 0.5 | 38.3 ± 0.5 |
| ReactiveGWM | 0.376 | 0.439 | 95.0 | 100.0 | — | 63.7 ± 2.1 | 59.7 ± 0.9 |
| StatePlay | 0.378 | 0.424 | 92.5 | 95.0 | 0.947 | 82.3 ± 0.5 | 78.3 ± 0.5 |
- Visual realism is not enough. Zero-shot baselines remain near chance on mechanics fidelity despite generating plausible gameplay: all score below 50% under Gemini-3.1-Pro, and the best GPT-5.5 score is 53.0%.
- Explicit state modeling improves mechanics fidelity. StatePlay achieves 0.947 state alignment and improves mechanics fidelity by 18.6 percentage points over the strongest stateless baseline under both visual judges, while maintaining competitive visual quality and action control.
Table 2. Architecture and State-Loss Ablation
MoT coupling with regression gives the strongest mechanics fidelity and state alignment.
| Coupling Style | State Loss | Visual Quality | Action Control (%) | State Alignment ↑Compares predicted timer, HP, and skill-meter traces with their targets using normalized error, then reports 1 − average distance. | Mechanics Fidelity (%) (n=3) | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Shared | MoT | FM | Regression | SSIM ↑Compares generated frames with aligned and cropped game-engine references for structural similarity, including layout, character shape, and local structure. | LPIPS ↓Compares generated and reference frames using deep visual features to measure perceptual distance. | Move‑Acc ↑Tracks the generated character with SAM 2.1 and Grounding DINO, then checks whether normalized displacement matches the commanded direction. | Att-Acc ↑Uses ClipAttackNet, a six-class frame-level attack classifier, to check whether generated attacks match the input command. | Gemini-3.1-Pro ↑Evaluates each generated video directly against reference images and target mechanic labels. | GPT-5.5 ↑Evaluates 24 uniformly sampled frames from each video against the same references and target mechanic labels. | |
| ✓ | ✓ | 0.309 | 0.541 | 75.0 | 90.0 | 0.804 | 52.0 ± 0.9 | 49.7 ± 1.2 | ||
| ✓ | ✓ | 0.355 | 0.450 | 97.5 | 100.0 | 0.943 | 64.7 ± 2.1 | 67.7 ± 0.5 | ||
| ✓ | ✓ | 0.363 | 0.430 | 85.0 | 71.7 | 0.845 | 60.7 ± 1.2 | 62.0 ± 1.4 | ||
| ✓ | ✓ | 0.378 | 0.424 | 92.5 | 95.0 | 0.947 | 82.3 ± 0.5 | 78.3 ± 0.5 | ||
- Coupling design. MoT improves mechanics fidelity by 17.6% over a shared backbone while preserving specialized representations and exchanging cross-modal information.
- State loss. Regression outperforms flow matching by 12.1% in state alignment and 21.6% in mechanics fidelity for compact, rule-governed states.
Limitation and Future Work
As publicly available datasets with synchronized state, frame, and action remain limited, we focus our initial evaluation on Street Fighter 3. Broader validation across game genres requires larger and more diverse state-aware datasets. Given the benefits of state modeling for mechanics-consistent generation, we hope this work encourages the development of datasets with explicit state annotations across a wider range of games, enabling state-aware game world models to scale and generalize across diverse rule systems.
BibTeX
@article{stateplay2027,
title = {StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation},
author = {Lin, Zijun and Wang, Zeqing and Tan, Cheston and Wen, Bihan and Jin, Yeying},
journal = {Under Review},
year = {2027}
}Paper and code links will be updated upon release.