PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation

## The Problem: Multi-View Hallucinations Break Robot Learning

Current world foundation models—neural simulators that predict future video frames—work well in single-camera setups but fall apart when robots use multiple cameras simultaneously. Robots need egocentric (first-person), eye-to-hand, and wrist-mounted views to manipulate objects safely, but existing multi-view world models simply stack different camera feeds without understanding how they relate geometrically. This causes objects to drift differently across views, depths to contradict each other, and textures to misalign. The result: the model's predictions become unreliable guides for robot control.

## How PAIWorld Fixes It

The authors' insight is that two fixes must work together: (1) explicit communication between views, and (2) actual 3D geometric reasoning.

**Geometry-Aware Cross-View Attention** creates a direct pathway between different camera views during the model's prediction process. Instead of processing each view independently and concatenating them later, this mechanism lets the model ask "what does that object look like from the other camera?" before generating predictions. This cross-talk catches inconsistencies early.

**Geometric Rotary Position Embedding** encodes actual camera geometry into the attention mechanism—specifically, camera ray directions (which pixels point where in 3D space) and the camera extrinsic poses (how cameras are positioned relative to the scene). Rather than treating spatial positions as abstract coordinates, the model learns that certain pixel locations correspond to specific 3D rays in the world.

**Latent 3D-REPA** is the technical backbone ensuring 3D consistency. The authors distill features from a frozen 3D foundation model—a neural network already trained to understand 3D structure—into the diffusion-based world model's latent space. This acts as a 3D-aware constraint: the world model's internal representations must respect actual 3D geometry, not just match pixels.

All three components sit inside a diffusion-transformer architecture (a type of foundation model that generates images by iteratively denoising noise into realistic predictions).

## Why It Matters

PAIWorld ranked 1st on the WorldArena leaderboard and 2nd on AgiBot-Challenge2026, both benchmarks for robotic manipulation in simulation. More importantly, the 3D consistency unlocks downstream applications: the model can guide model-based planning (where robots reason about future states before acting), support world action models (predicting how robot commands change the scene), and improve multi-view policy training (teaching robots to act from multiple camera angles).

For roboticists, this means simulation now reliably captures how objects move and look across the different viewpoints a real robot sees. That reliability translates to policies that transfer better from simulation to physical robots—the long-standing bottleneck in robotic learning. The explicit geometric reasoning, rather than brute-force pattern matching, likely explains why the approach generalizes: it learns principles of 3D consistency rather than memorizing view pairs.