FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

The Problem: World Models Fail in Chaotic Cities

Existing world models—neural networks trained to predict future video frames—work reasonably well on orderly highway driving or controlled lab settings. But they collapse when applied to crowded, chaotic streets in Global South cities: dense pedestrian traffic with no lane markings, vendors blocking sidewalks, motorcycles weaving through buses, partial occlusion everywhere, and constant informal negotiation over space. The authors call this regime DENSEWORLD. Current Joint Embedding Predictive Architectures (JEPA)—a family of models that learn abstract representations without reconstructing raw pixels—compress the future into a single latent vector, which proves inadequate for capturing the intricate, heterogeneous interactions in these scenes.

The Core Insight and Method

FactorJEPA's key innovation is factorizing the monolithic future representation into three separate channels: layout (road geometry, static obstacles), entities (individual agents), and interactions (how agents influence each other). Instead of one latent bottleneck, the model maintains dedicated subspaces for each factor, connected with a visibility gate that explicitly handles partial observability—crucial when pedestrians vanish behind parked cars or buses.

The visibility gate is the critical detail: it prevents the model from cheating by reconstructing occluded agents from context alone. Separated subspaces also block "shortcut" learning where the interaction channel could simply copy information already present in the entity or layout channels. This architecture forces the model to genuinely learn how agents negotiate shared space rather than memorizing static scene patterns.

Why It Matters

The authors created the first large-scale benchmark for this problem: DENSEWORLD, comprising 1,000 hours of video across 22 Global South cities (drive-throughs, walk-throughs, aerial views). This dataset captures the visual complexity that existing urban-driving benchmarks (usually highway-focused) ignore entirely.

FactorJEPA demonstrates measurable improvements across four metrics:

• Future-frame L1: Cleaner predicted frames (lower reconstruction error)
• Causal L1: Predictions that shift correctly when agents are intervened (moved), proving the model tracks causality, not just correlation
• Mask-ratio slope: Graceful degradation as visual information is removed—staying accurate even with significant occlusion
• Motion-information trade-off: A reproducible tension between motion plausibility and information preservation, surfaced as a diagnostic tool

Results hold across two model scales (2B and 1B parameters), with high correlation (ρ = 0.895–0.978), indicating the findings are robust, not artifacts of a single architecture.

The Practical Payoff

This work matters because autonomous vehicles and robotics increasingly operate in dense, chaotic urban environments—especially in the Global South, where infrastructure is informal and agent behavior is heterogeneous. A world model that can jointly predict layout, agents, and their interactions provides a richer, more controllable foundation for planning and reasoning. By open-sourcing the DENSEWORLD-115k dataset and trained checkpoints, the authors create infrastructure for future research in an understudied but economically important regime.