WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations

The Core Claim

Realistic video datasets for training world-building AI models are missing crucial annotations. WorldRover solves this by creating a synthetic video engine that generates long, explorable sequences paired with ground-truth camera motion, 3D geometry, optical flow, and action signals—all recorded simultaneously from the same rendered scene. This turns what's normally hard to capture in real video into a scalable, controllable generation problem.

Why This Matters

Training models to generate or reconstruct interactive 3D worlds requires more than just RGB frames. They need to understand camera movement, where objects sit in 3D space, how pixels correspond across time, and (for interactive applications) what actions produced each motion. Real-world video captures RGB reasonably well but struggles with dense geometry and long-range correspondence—you either estimate these imperfectly or use expensive equipment. Rendering bypasses this: a simulator knows exact depths, trajectories, and matches.

The catch: existing synthetic datasets rarely pack all these signals into one coherent resource. They often skip certain modalities or don't support the multi-view, multi-state replays needed for robust learning. WorldRover addresses this by building a turnkey pipeline.

How It Works

WorldRover-Engine is built on Unreal Engine. It takes an artist-designed scene and executes pre-planned exploration routes—minute-long traversals of the environment. As the engine renders, it records everything simultaneously: RGB video, metric depth maps, precise camera trajectories, and action signals (derived from trajectory changes).

The key innovation is replay flexibility. Once a route is rendered, it can be re-rendered from multiple viewpoints and under different conditions—all without re-planning:

• First-person: the original explorer's view
• Third-person: a detached camera watching the character move
• 360 panoramic: full-surround video from a fixed location

Each variant preserves the same underlying route and geometry. The engine can also toggle environmental states (lighting, weather) or switch to neutral white materials, isolating geometry without appearance variation.

This multi-view, multi-state capability is crucial: it lets models train on diverse perspectives and conditions from a single exploration, multiplying data efficiency.

What You Get: WorldRover-10M

The authors use this engine to build WorldRover-10M, a dataset pairing RGB sequences with:

• Metric depth (true 3D shape)
• Camera trajectories (ground-truth motion)
• Action signals (what caused motion changes)
• Dense optical flow (per-frame pixel correspondence)
• 2D/3D point tracks with visibility (which 3D points appear where over time)
• Character trajectories separate from camera motion (for third-person variants)

This richness is uncommon. Most video datasets offer one or two of these; WorldRover bundles them at scale, making it practical to train models that build and maintain coherent world representations—a critical capability for embodied AI and interactive content generation.

The scalability angle matters too: once the pipeline runs, generating new explorations becomes a data-generation problem, not a capture problem. This could democratize access to training signals previously available only to labs with specialized equipment or simulation budgets.