Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training

The Core Problem: Game UIs Are Ruining World Model Training

Video games offer unlimited training data for world models—diverse environments, complex physics, rich interactions—but there's a catch: gameplay videos are cluttered with heads-up displays (HUDs), health bars, minimap icons, and menu overlays that have nothing to do with the actual game world. These screen-space artifacts introduce noise that confuses models trying to learn how the world actually works. A world model trained on UI-covered footage learns spurious correlations (like "red numbers disappear when I jump") instead of true physical dynamics. The authors' core claim: automatically remove these UIs from gameplay videos at scale, and world models trained on the cleaned footage perform significantly better.

How Game2World Works

The authors built a three-part system:

1. UI Asset Library & Taxonomy. They manually curated 5,132 verified UI elements from 1,010 gameplay frames across 21 categories (health bars, minimaps, buttons, etc.). This taxonomy lets the system recognize what is and isn't part of the game world.

2. Synthetic Data Generation. Using the asset library, they created 96K paired videos: clean gameplay footage synthetically overlaid with realistic UI elements in temporally coherent ways. This gives them ground truth for training—they know exactly what was added and where.

3. GameCleaner Model. Unlike older "mask-based" methods that require pixel-level annotations of which regions to remove, GameCleaner is mask-free. It combines multimodal semantic understanding (learning what text and icons mean) with video editing capabilities to directly identify and remove HUD elements while preserving the underlying scene and temporal continuity.

Why It Matters & The Results

The payoff is quantified: world models trained on UI-removed gameplay achieve 6.83% improvement in VideoReward (a metric measuring how well models predict future frames) compared to models trained on raw footage. On controlled synthetic benchmarks, GameCleaner removes UI with 95.36% accuracy (AAR—Average Accessibility Ratio), outperforming the strongest temporal mask baseline by 57.3%. On real, messy in-the-wild videos from 303 games, it achieves 80.05% accuracy while preserving 99.8% of the background scene.

The broader significance: gameplay videos are freely available everywhere on the internet. By automatically cleaning them, the authors unlock a massive, scalable source of high-quality training data for world models. They've released their UI element library, 96K synthetic paired videos, and GameCleaner to enable future research. This shifts video world modeling from lab-curated datasets toward real-world internet-scale data—a necessary step if these models are ever to power embodied AI systems.