Gaze Target Estimation Anywhere with Concepts

The Problem with Current Gaze Estimation

Estimating where a person is looking in a photo sounds simple but is genuinely hard. Existing methods break the task into rigid stages: find the head, estimate pose, predict gaze direction. Each stage can fail independently, and errors compound. Worse, these approaches force you to specify which person you care about through technical inputs like bounding boxes—clunky for real applications. The authors argue that natural language prompting—saying "the boy in the red shirt"—would be far more practical and flexible.

The Core Contribution: Promptable Gaze Estimation

The authors introduce Promptable Gaze Target Estimation (PGE), which flips the pipeline. Instead of requiring explicit head boxes and pose as separate preprocessing steps, the model takes a single image plus a natural language or visual prompt identifying the subject, and directly outputs where that person is looking. This is end-to-end: localization and gaze prediction happen together, not sequentially.

The key innovation is treating gaze estimation as a concept-driven task. A user can specify the subject via text ("the person on the left wearing blue"), spatial coordinates (a point on the image), or visual features. The model learns to ground these prompts and compute gaze from the same learned representation, avoiding fragile multi-stage cascades.

The Method and Dataset

The authors built GazeAnywhere, a transformer-based detector that fuses frozen feature encoders (likely pretrained vision models) to jointly solve three problems: which person to analyze, whether their gaze target is in or out of frame, and a heatmap showing where they're looking. The architecture sidesteps the need for intermediate pose estimation entirely.

To train and evaluate PGE, they created Gaze-Co, a dataset of 120K prompt-annotated image pairs. A "data engine" scaled annotation by leveraging synthetic and real imagery with systematic labeling of gaze targets, subject location, and natural language descriptions.

Why This Matters

The shift from rigid pipelines to flexible prompting has proven invaluable in other vision tasks—think image segmentation with "Sam" or visual question answering. Gaze estimation lagged behind because no one had framed it this way or built the infrastructure to scale it.

By eliminating error cascades, PGE is more robust: if pose estimation fails in the old pipeline, gaze fails too. Here, the model sees the full image and refines both localization and gaze jointly. On multiple benchmarks, GazeAnywhere achieved state-of-the-art results, and critically, it generalized to a difficult real-world clinical dataset—suggesting the approach is genuinely more reliable out-of-domain.

The practical upside is clear: researchers, applications, and end-users can specify gaze subjects conversationally or by pointing, rather than wrangling bounding boxes. The authors open-sourced the code and dataset, lowering the barrier for downstream work.