GameDirectorDecoupling Gameplay Logic from Rendering
for Player-Configurable Game World Models

Zijun Lin1,2,4,* Zhiyang Deng1,3,* Yuzhe Wu1,3 Bihan Wen2 Yeying Jin1,3

  1. Tencent
  2. Nanyang Technological University
  3. National University of Singapore
  4. Centre for Frontier AI Research, A*STAR

* Equal contribution.

Configure the game. Direct the action. Generate the world.

Explore the visualizations
Game worlds
3
No Rest for the Wicked · Vampire · Hollow Knight
Characters
13
5 players, 8 bosses
Initial states
Any
HP, attack strength and skill meters — set by the player
Mechanics fidelity
99.6%
vs. 21–34% for end-to-end baselines
Decision quality
+39.9%
Improvement over the best end-to-end baseline
01

Abstract

Recent game world models support realistic visual simulation and interactive gameplay based on player inputs. However, they typically learn environment dynamics from pixel-level supervision, jointly modeling perception, memory, state transitions, and rendering within a single end-to-end framework. While this design enables open-ended, action-controllable generation, it still falls short of delivering a complete gameplay experience. Games are governed by explicit mechanics, such as health deduction, skill activation, combat rules, and termination conditions. These mechanics depend on precise and consistent state transitions that generative models alone cannot reliably enforce. In contrast, traditional game engines can guarantee such mechanics through hard-coded rules, but provide limited flexibility for player-driven creation. To bridge these paradigms, we introduce GameDirector, the first agentic framework that decouples rule-based gameplay logic from visual rendering. Given player-defined configurations, the framework acts as an intelligent director that interprets visual observations, updates game states, tactically controls NPCs, and enforces gameplay rules. It then translates these decisions into text prompts that guide the video world model to render the resulting gameplay. This separation allows players to configure characters, states, and rules much like a game developer while preserving coherent game mechanics. Experiments on three games, using data collected by our automated gameplay agent, show that GameDirector achieves accurate state tracking, reliable rule following, and improves boss action quality by more than 39.9% over various end-to-end game world model settings. Overall, by externalizing player-controllable game logic, GameDirector establishes an effective middle ground between hard-coded simulation and generative modeling, enabling more flexible and closed-loop gameplay experiences.

GameDirector overview: configurable initialization, an agentic director, and a world model renderer bridge flexible generation and reliable mechanics.
Overview of GameDirector. Our framework bridges hard-coded rules and generative modeling to enable player-configurable, state-aware, and mechanics-consistent gameplay.
02

Dataset Construction

An automated gameplay agent collects synchronized gameplay videos, player controls, boss actions and engine-internal states across No Rest for the Wicked, Vampire and Hollow Knight. The dataset contains 51,758 clips spanning 13 characters—5 players and 8 bosses—with 300 clips reserved for evaluation (100 per game).

Dataset construction pipeline: automated keyboard and mouse control, gameplay recording, time-aligned spatial and combat state collection, then filtering, synchronization and annotation.
Spatial states (positions, distances and angles) and combat states (health points, skill meters and executed skills) are aligned on a shared timeline. Irrelevant segments are filtered out, and raw control and action logs become structured prompts for gameplay generation and visual understanding.
03

Method

The player sets the stage. The director decides what happens. The painter brings it to life.

GameDirector architecture: player configurations feed visual understanding, explicit state tracking, a 2B action planner and rule enforcement; validated prompts guide video generation in a closed loop.
01

Configure

Select the scene, characters and skills. Set health points, attack strengths, skill meters and skill-specific rules.

02

Direct

AttackNet and SituationNet interpret combat. Explicit states, a 2B language-model planner and rule enforcement determine valid actions.

03

Generate

A Wan2.2-TI2V-5B world model renders structured prompts. Visual feedback closes the loop between game logic and generation.

04

Visualization

01

Player-configurable initialization

The same director narrates two different starting configurations — health, attack strength and skill meters — while the rendered gameplay stays faithful to each state.

02

Cross-character skill transfer

A skill learned on one boss is reassigned to another. The director keeps the action legible while the world model preserves each character's appearance.

Hornet → Broken Vessel / Dung Defender
03

Adaptive boss action planning

Given the spatial context, the 2B action planner selects a skill consistent with distance, facing and cooldown rules — and the render follows through.

05

Quantitative Results

GameDirector's agentic control outperforms every end-to-end game world model baseline on state alignment, mechanics fidelity and boss decision quality.

State alignment
100%

Tracked states always follow game mechanics — HP and skill meters update exactly as they should.

Mechanics fidelity
99.6%

VLM-judged adherence to termination and cooldown rules, best across both GPT-5.5 and Gemini judges.

Decision quality
93.0%

Boss actions fit the spatial context and available skills, up to +45.8 points over the best baseline.

Visual understanding
>90%

AttackNet's hit detection and SituationNet's spatial classification both exceed 90% accuracy across all three games.

Table 1. Comparison of different state modeling and boss control strategies. Each setting adopts a representative method with minimal adaptation, and all methods are trained using the same dataset. We use N = 3 VLM evaluations and “--” denotes metrics that are not applicable to a given method.
State ModelingBoss Control
SettingNonePredictiveImplicitExplicitGPT-5.5GeminiGPT-5.5Gemini
(1) Matrix-Game--25.1± 0.725.0± 0.328.1± 0.926.4± 0.2
(2) ReactiveGWM--21.4± 0.221.4± 0.148.5± 0.947.2± 0.3
(3) WildWorld70.3834.3± 0.534.1± 0.427.1± 0.426.6± 0.3
(4) StatePlay72.0128.1± 0.328.2± 0.144.3± 1.044.6± 0.1
GameDirector (Ours)AgenticAgentic100.098.5± 0.299.6± 0.288.4± 0.793.0± 0.6
Table 2. Performance of AttackNet and SituationNet in Visual Understanding across three games.
AttackNetSituationNet
Game
Game N0.847690.530.355610.9189.2987.0686.95
Game V0.913592.480.18762.2296.7296.3196.31
Game H0.887196.110.26376.3996.4996.4996.49
Average0.890692.940.26906.5194.1793.2993.25

Game N: No Rest for the Wicked · Game V: Vampire · Game H: Hollow Knight.