SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation

The Core Claim

Spatial understanding—reconstructing 3D shapes, finding correspondences between images, and reasoning about spatial relationships—doesn't need separate specialized models. A single multimodal generative model can handle all three tasks better when trained jointly, because they share geometric knowledge about the same physical scenes. SPARGen demonstrates this by casting these three distinct problems as variants of the same generative task.

How It Works

The key insight is treating spatial problems as instruction-conditioned generation. Rather than building separate pipelines for 3D reconstruction, correspondence matching, and spatial reasoning, SPARGen unifies them under one framework:

The input-output translation: The model takes an image and a natural language instruction (e.g., "reconstruct this object in 3D" or "find dense pixel correspondences with this other image") and generates outputs as token sequences. For dense geometric outputs—like depth maps or correspondence fields—the model generates them in image-aligned forms, preserving spatial structure.

Compact serialization: Instead of outputting raw point clouds or unstructured data, SPARGen serializes both structured outputs (coordinates, geometry) and linguistic information as tokens, the same currency that language models naturally work with. This lets the model treat 3D geometry, correspondence, and reasoning in a unified representational space.

Joint spatial supervision: Because all three tasks share geometric understanding of the same scene, the training process uses spatial supervision (ground truth 3D shape, correspondences, or relations) to update shared representations simultaneously. A single forward pass through the generative model can improve performance on all three downstream tasks.

Why It Matters

Breaking the task-specific wall. Most current approaches optimize separate architectures for each spatial problem—dedicated 3D reconstruction networks, correspondence matchers, spatial reasoning modules. This prevents knowledge from flowing between complementary tasks. SPARGen shows that spatial understanding is fundamentally unified; training on one task helps the others.

Efficiency and simplicity. A single native multimodal generative model eliminates the need for external geometric modules or post-processing pipelines. This reduces engineering complexity and memory overhead compared to stacking multiple specialized models.

Competitive performance across tasks. The authors validate SPARGen on benchmarks for all three capabilities—3D reconstruction, dense correspondence, and spatial reasoning—and report competitive results within a single framework. This isn't a trade-off where one task suffers; the unified approach remains effective across heterogeneous spatial problems.

Generalization potential. By grounding spatial reasoning in the same generative process that produces geometric outputs, the model learns more transferable representations. Instructions guide the same underlying spatial understanding, making the model more flexible for novel spatial queries.

The paper essentially argues that modern multimodal generative models—which already excel at instruction-following and dense prediction—are a better substrate for spatial AI than task-specific architectures. By serializing geometry as tokens and treating it like language generation, SPARGen demonstrates that spatial perception and reasoning are naturally compatible problems within a unified framework.