Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

The Real Problem Isn't Seeing—It's Choosing What to Trust

Multimodal Large Language Models (MLLMs) pair images with text, leveraging both visual input and learned language knowledge. They work well on many tasks, but fail spectacularly when the two sources conflict—say, when an image shows a blue object but the model's training data insists it should be red. The authors' central claim: the bottleneck isn't perception; it's post-perceptual decision-making. MLLMs encode visual details but struggle to reliably prioritize visual evidence over ingrained language priors.

This is a significant reframing. It means fixing MLLMs won't require redesigning vision encoders—the actual visual information gets through. Instead, the problem sits downstream, in how the model weighs and acts on that information.

How They Diagnose the Problem

The authors use two complementary approaches.

First, image reconstruction. They extract the final-layer image tokens from frozen MLLMs and attempt to reconstruct visual attributes (spatial layout, color, count, size, weight) directly from those tokens. Success here proves that coarse-grained visual information survives encoding. It does—which immediately implicates downstream mechanisms in the failures.

Second, WhatIfVis benchmark. They built a dataset of questions spanning five dimensions (spatial-temporal, color, count, size, weight) where the correct answer could plausibly come from either the image or the model's prior knowledge. For example: "Is the cat sitting or standing?" (visual) versus "Does salt dissolve in water?" (prior knowledge, image-agnostic). By varying instructions—"rely on the image" versus "rely on your knowledge"—they measure how well the model switches between the two sources.

The results reveal instability. Vanilla models ignore explicit instructions about whether to use visual evidence. Fine-tuning on WhatIfVis improves this controllability and even generalizes to new domains.

The Steering Vector Solution

The authors then use activation patching—a technique that traces which internal layers of the model control the vision-versus-prior trade-off. They discover this trade-off localizes at specific depths (architecture-dependent across six models tested). Crucially, they show this trade-off can be steered along a learned vector without explicit instructions. Applying this vector improves the model's reliance on visual context automatically, even in vanilla (unfinetuned) versions.

This is the actionable insight: the vision-prior competition isn't a mysterious black box. It's encoded in the model's learned representations and can be nudged.

Why It Matters

This work reorients how we think about MLLM failures. Rather than assuming poor vision encoders or fundamental architectural flaws, the authors demonstrate that visual information arrives intact but gets misweighted downstream. That's good news—it suggests practical fixes are possible (fine-tuning, steering vectors) without major redesigns.

For practitioners building or deploying MLLMs, the implication is clear: if your model hallucinates or ignores images, the problem likely isn't what it sees, but what it chooses to believe. The paper provides both diagnostic tools (WhatIfVis) and control mechanisms (steering vectors) to address that choice.