ReferTrack: Referring Then Tracking for Embodied Visual Tracking

The Problem with Current Embodied Tracking

When a robot needs to follow a target described in natural language ("track the person in the red jacket"), most recent systems use vision-language-action (VLA) policies that try to do everything at once: identify the target and plan movement in a single reasoning step. The trouble is that this reasoning happens in abstract, internal feature spaces that are hard to supervise and don't cleanly map back to what the camera actually sees. The result: weaker target identification and harder debugging.

A Two-Stage Approach Grounded in Images

ReferTrack splits the problem into two explicit steps, both anchored to concrete image observations. First, the model selects the target from a set of detected bounding boxes in the current frame—a discrete, supervise-able decision. Second, it plans tracking waypoints using that grounded choice as a condition. This separation lets the authors use standard object detection losses for target ID and waypoint regression for navigation, making each stage interpretable and trainable.

The key innovation is handling temporal continuity: a robot following a target needs to remember where the target was, not just where it is now. ReferTrack maintains a sliding-window queue of previously selected bounding boxes from past frames. Instead of burying this history in a latent representation, the model injects geometric features from these old boxes directly as tokens (called TVBI tokens—temporal-viewpoint-bbox indicators) that condition the visual encoder. This preserves motion cues in an explicit, interpretable form.

Results and Real-World Validation

On EVT-Bench, a standard benchmark for embodied visual tracking, ReferTrack achieved state-of-the-art results with a single forward-facing camera:

• 89.4% success on single-target tracking
• 73.3% on distracted splits (target partially obscured or occluded)
• 74.1% on ambiguity splits (multiple similar objects in scene)

These numbers match or exceed several baselines that use multiple cameras, suggesting that explicit grounding in image space is more efficient than implicit multi-view fusion.

The authors also co-trained on a custom Refer-QA dataset to sharpen target identification—a dataset of question-answer pairs that teach the model to distinguish targets by visual and contextual attributes. This boosts performance on identification-heavy scenarios.

Crucially, the method transferred to real robots. Deployments on both legged and humanoid platforms showed robust sim-to-real generalization, a notoriously brittle step for learned visuomotor policies. The fact that explicit image-grounded decisions transfer better than latent reasoning suggests the approach is capturing more robust, generalizable features.

Why It Matters

This work challenges the "end-to-end everything" trend in embodied AI. By forcing the model to make explicit, supervise-able decisions tied to image observations, ReferTrack becomes more interpretable and easier to debug—critical for safety-critical robotics. The single-camera performance matching multi-camera systems also hints that information bottleneck isn't the limiting factor; clarity and grounding are.