From Pixels to States: Rethinking Interactive World Models as Game Engines

The Real Problem with AI Game Engines

Recent video generation models can predict what happens next in a game based on player actions—impressive feats of pattern matching. But they're missing something fundamental: they don't actually understand the game state. They generate pixels without tracking whether a player's health changed, whether a door unlocked, or whether consequences persist across time. The authors argue that to build genuine interactive worlds, AI needs to adopt what traditional game engines have always done—maintain an explicit game state that evolves according to rules, then render observations from that state. Without this, generated worlds look plausible frame-to-frame but fall apart under extended play.

Four Dimensions of the Problem

The paper organizes the challenge into four axes. Player action control asks: how do we let players meaningfully steer the world? Game state dynamics requires tracking what changes in response to actions—a player shoots an enemy, the enemy's health decreases. State-observation persistence means changes must stick around; a destroyed wall stays destroyed. Real-time generation demands the system runs fast enough to feel interactive, not cinematic.

Current video generative models optimize for visual quality but ignore most of these constraints. They predict the next frame conditioned on the previous frames and action, without maintaining explicit state variables. Over long sequences, incoherence accumulates—the same action might produce different outcomes depending on subtle pixel variations in prior frames. In contrast, traditional game engines enforce consistency by separating logic (state updates via rules) from rendering (visual output from state).

The Data Contribution

To ground this analysis, the authors built a large-scale dataset from Black Myth: Wukong, a complex commercial game. They collected over 90 hours of gameplay with frame-aligned data: the actions players took, the ground-truth internal game state at each moment, and the rendered observations. This isn't just raw video—it includes structured and semantic annotations, making it possible to study how actions map to state changes and how state determines what the player sees.

This is the critical missing piece in prior work. Most research uses synthetic environments or simplified games where state is easy to extract. Real game engines are black boxes; getting frame-by-frame ground-truth state from a commercial title is laborious and rare. Having this resource enables new research on state-aware world models—systems that learn to predict state transitions before generating pixels.

Why It Matters

The paper doesn't present a single breakthrough method. Instead, it provides conceptual clarity and infrastructure. By mapping existing approaches onto the action-state-observation loop, it shows where current methods fall short and suggests a more principled direction: learn what actually happens in the game world (state changes), not just what it looks like. This reframing matters because it sidesteps the pixel-generation arms race and points toward systems that could generalize beyond training data, handle novel player inputs, and maintain logical consistency over hours of gameplay—the actual requirements of an interactive game engine.