See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

The Core Problem: Seeing in One Frame, Acting in Another

Vision-language-action (VLA) models—neural networks that take images and text instructions to produce robot control commands—face a hidden geometric mismatch. Robots define their actions (grasp here, move there) in their own 3D coordinate system, but most VLAs learn from camera views that don't align with that frame. When the camera position is fixed during training, the model can memorize the visual-to-action mapping. But real-world datasets combine demonstrations from different camera angles, forcing the model to solve a much harder problem: generalize the same action across viewpoints without explicit 3D geometry.

The authors propose a simple fix: robot-centric pointmaps. Instead of feeding the model an RGB image, feed it an image where each pixel stores the 3D coordinates of the scene point it sees—all expressed in the robot's coordinate frame. This keeps the familiar grid structure (H × W pixels) that pretrained 2D models expect, while encoding the geometry needed to understand the scene from the robot's perspective.

How Pointmaps Work

A pointmap is generated from standard sensor data: depth images and camera calibration parameters. For each pixel, the model computes the 3D position of the scene point it observes, then transforms those coordinates into the robot's frame. The result is an image-like array where red, green, and blue channels encode x, y, z robot coordinates instead of color. Depth and camera intrinsics make this transformation straightforward; no learning required.

Crucially, pointmaps preserve what makes pretrained vision models useful: they're still dense 2D grids. Existing VLA architectures need almost no changes—swap RGB inputs for pointmap inputs and train. The authors demonstrate this on two existing models, pi0.5 and SmolVLA, showing consistent improvements with minimal code modification.

Why It Matters and What the Results Show

On RoboCasa—a large-scale simulation benchmark with diverse camera placements—pointmaps outperform both RGB-only baselines and other 3D-aware approaches. The gains are clearest in real-world experiments: when the camera moves to a position never seen during training, an RGB-only policy degrades, but the pointmap-based policy maintains performance. This is exactly the scenario where fixing the frame mismatch pays off.

The insight cuts to a fundamental tension in robot learning: pretrained 2D vision models are powerful and well-understood, but they operate in a space (image pixels) divorced from where actions live (robot coordinates). Rather than redesign the whole pipeline for 3D, pointmaps translate the geometry problem into a form the existing 2D machinery can solve. It's a format bridge, not an architectural overhaul.

The work addresses a real bottleneck for scaling VLAs to diverse real-world deployments where camera rigs vary. By making explicit the 3D information already implicit in depth and calibration data, pointmaps let models learn more robust, generalizable policies without abandoning the efficiency and availability of pretrained 2D encoders.