WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning

The Core Finding

Robots need to understand sequences of observations to act effectively. WorldToken proposes a simple but specific way to organize this information: bundle all sensor inputs at each decision point—images from multiple cameras, joint positions, task descriptions—into a single "world token," then feed these tokens chronologically through a causal Transformer. On 23 household manipulation tasks (RoboCasa), this approach achieves 59.45% success rate with a modest 85.3M-parameter model trained on just 2,900 synthetic demonstrations per task, using only a frozen CLIP encoder as pretrained weights.

The critical finding is about temporal context: when the model can only see one or two recent timesteps instead of the full history, performance degrades across all 50 policies tested. On a separate benchmark (RMBench), visibility that drops from 146 seconds to 8 seconds collapses success from 95% to 28%. Even in an extended scenario with 850+ seconds of rollout, maintaining access to full history proved essential.

How It Works

The method stacks three components:

Input fusion: At each timestep, the policy ingests multiview images (from cameras at different angles), proprioceptive data (arm joint angles, gripper state), and task conditioning (text describing what to do). Rather than processing images separately or interleaving sensor modalities, WorldToken compresses all of this into a single learned vector—the "world token."

Temporal modeling: A causal Temporal Transformer then processes the sequence of world tokens. Causal means the model at timestep t can only look backward, never forward, which is required for closed-loop control.

Action generation: Instead of predicting single actions, a diffusion-based action head generates chunks (multiple timesteps of actions at once), which is a common technique in recent imitation learning to improve stability.

The policy is trained end-to-end on 2,900 demonstrations per task using standard imitation learning loss, with only CLIP frozen (used to encode task text).

Why It Matters

The results establish two practical insights. First, the data-scaling and model-scaling sweeps show the expected patterns: more target-domain demonstrations help, but returns diminish past moderate model sizes (suggesting 85M parameters is reasonable). This is useful for practitioners deciding how much data to collect and how large their policy should be.

Second, and more important: the temporal context experiments reveal that robots do need to "see" their recent history. This isn't a surprising philosophical claim—it's an empirical characterization. Truncating history to recent timesteps consistently hurts. The 146-to-8-second drop in RMBench is especially striking and suggests that even for relatively simple block manipulation, context beyond a few seconds matters.

However, the authors are careful to note what they don't claim: they don't show WorldToken beats alternative designs (like vision-first or action-first sequences), and they don't isolate which components—the Transformer, diffusion head, CLIP backbone, or the token design itself—drive the gains. The paper is scoped as an empirical characterization of one complete recipe, not a systematic comparison.

This matters because sequence modeling for robotics is still unsettled. WorldToken offers a concrete, reproducible instantiation with honest ablation constraints that future work can build on or contradict.