3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

The Core Problem

Most robot manipulation systems that combine vision and language split the task into two parts: a high-level planner decides what to do, and a low-level controller decides how to move. Recent systems use a vision-language model (VLM) to output a sequence of 2D waypoints on the image plane—but this creates a critical mismatch. The low-level controller operates in full 3D space (using point clouds from depth sensors), so it needs 3D coordinates. When given only 2D points, the system guesses the depth by looking at whatever surface happens to be directly below each point in the image. This produces geometrically nonsensical trajectories, especially when objects are stacked or when the scene changes.

How 3D HAMSTER Works

The authors propose a straightforward fix: make the planner output 3D trajectories directly. They augment a standard VLM with two additions: a depth encoder that processes depth images, and a training objective that forces the model to predict depth at each waypoint. The planner now outputs full 3D coordinates rather than 2D pixels.

The depth encoder extracts spatial structure from depth maps—information the vanilla VLM never sees. The dense reconstruction objective ensures the model learns to predict depth reliably across the entire image, not just at isolated points. These enriched 3D waypoints then feed directly into an existing point-cloud-based policy that controls the robot's arm.

The framework is hierarchical: the VLM stays responsible for understanding language and scenes, while the policy stays responsible for smooth, feasible motion. The depth module acts as a bridge, translating from the VLM's visual understanding into the 3D metric space where the controller lives.

Results and Impact

Testing spans three settings—predicting 3D trajectories, simulation, and real robots. On all three, 3D HAMSTER beats proprietary VLMs (like GPT-4V) and 2D-guided baselines. The gains are largest when conditions shift visually (e.g., different lighting or textures) or semantically (unseen object types, novel spatial arrangements, language the model hasn't encountered). This matters because real-world deployment demands robustness to variation.

The insight is simple but practical: forcing the planner to commit to 3D geometry early—rather than punting the depth problem downstream—tightens the coupling between planning and control. It removes a source of error and lets each stage of the pipeline work in the space where it's most natural.

The authors provide a project page with reproducible details, suggesting the method is designed to be adopted by future systems in this hierarchical planning-and-control paradigm.