Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings

## The Core Insight

Standard image retrieval systems treat each training pair as a single scalar signal: two images either belong to the same class or they don't. This crude supervision ignores *why* images are similar or different. The authors show that frozen multimodal large language models (MLLMs)—systems trained on both images and text—can articulate the specific visual attributes (plumage patterns, wheel design, fuselage shape) that distinguish or unite image pairs. By using those attribute-level judgments as a training signal, they push vision encoders to encode the right visual details rather than crude class-level patterns.

## How SAGA Works

The framework has three components working in tandem:

**Attribute-aware reward signal.** An MLLM (frozen and discarded after training) receives two images and lists which visual attributes match or differ between them, then predicts if they share a class. The authors use Group Relative Policy Optimization (GRPO), a reinforcement-learning technique, to reward the MLLM when its prediction is correct. The gradient flows backward through the MLLM's reasoning—its reasoning about fine-grained attributes—but lands on the vision encoder's token embeddings that fed into the MLLM in the first place. This reshapes the encoder to expose those exact attributes, replacing uniform "same/different" signals with attribute-resolved supervision.

**Attention anchoring.** An auxiliary loss pulls the encoder's embedding toward the specific tokens the MLLM attended to when making its prediction. This ensures the encoder's learned representation aligns with human-interpretable visual concepts the language model found relevant.

**Embedding geometry.** A standard metric-learning loss (e.g., contrastive or triplet) shapes the embedding space so that similar images cluster together in a way that aids nearest-neighbor retrieval.

The MLLM is a training-only oracle. Once the encoder is trained, it discards the frozen model and runs inference with just the vision encoder—matching the computational cost of existing baselines.

## Why It Matters

Fine-grained visual retrieval (distinguishing between bird species, car models, or aircraft variants) requires capturing subtle, discriminative details. Traditional class-label supervision provides no guidance on *which* details matter. By grounding training in language-based attribute reasoning from an MLLM, the authors sidestep the need for expensive attribute annotations while leveraging the rich semantic understanding embedded in pre-trained multimodal models.

The results validate the approach: on four standard benchmarks (CUB-200-2011 for birds, Cars-196 for vehicles, FGVC-Aircraft, and iNaturalist Aves), SAGA improves Recall@1—the fraction of correct results in the top-1 retrieval—by 3 to 6 percentage points over state-of-the-art methods. These gains come in zero-shot settings, meaning the encoder generalizes to unseen classes, a practical requirement for real-world deployment.

The insight generalizes beyond these niche domains: any visual retrieval problem benefits from encoding task-relevant attributes rather than brittle class boundaries. The method is computationally efficient and modular, making it applicable wherever frozen MLLMs are available.