Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Control

The Core Insight

World models—systems that imagine future video frames or trajectories—are typically valued for their output: the predicted future itself. This paper argues the real treasure is hidden inside: the intermediate computations the model performs while generating that future. Those internal states implicitly organize spatial layout, appearance, and object interactions at multiple levels of abstraction. The authors ask: can you bake this future-generative computation into a representation that operates on the present alone, without generating an actual future at test time?

Enfold does exactly that. By learning a representation that mimics the internal structure of a world model generator—but infers it from current visual input and language instructions—the approach eliminates the computational bottleneck of actually running the generator at deployment. The result: 3.7× faster action prediction (10.1× faster in a variant called Enfold-Flash) while maintaining strong control performance.

How It Works

During training, the authors use a world model generator that processes an observed future video and refines it step-by-step into a coherent trajectory. As the generator works, its hidden states at different depths capture progressively richer structure. These multi-level internal states become supervision signals for an encoder that operates only on the current observation plus a language instruction.

The learned representation is then fed into two pathways: (1) it conditions future generation when needed, and (2) it feeds into task-specific control heads that predict actions. Critically, task gradients don't backpropagate into the encoder—this prevents action-learning objectives from distorting the representation learned from the generator's internal structure.

At test time, no generator is run. The encoder ingests the current scene, outputs a predictive representation, and action heads immediately produce control signals. When the environment changes—even if a human intervenes—the representation adapts because it's inferred fresh from the new observation.

Why It Matters

The speedup is practical and substantial. In robotic control, latency directly translates to performance loss; 3.7–10× reductions unlock deployments previously ruled out by computational constraints.

Beyond speed, the approach reshapes how we think about world models. Generators are expensive to run but rich in structure; most prior work throws that structure away after reading off a rendered future or latent code. Enfold recovers it. The representation analyses confirm the method captures meaningful patterns: it suppresses visual distractions (nuisance variation) and emphasizes changes unfolding over longer time horizons—exactly what you'd want for planning.

The real-robot experiments and ablations on LIBERO and RoboTwin2.0 show the method isn't just fast; it generalizes. The ability to adapt to human interventions—rather than replaying a fixed imagined trajectory—suggests the representation genuinely encodes predictive structure, not memorized rollouts.

This reframes world model design: if the generator's internal computation is the asset, then distilling it into a current-only representation is a natural next step, trading the cost of imagination for the efficiency of amortized understanding.