See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Do Multimodal Models Actually Use the Sketches They Draw?

Recent multimodal large language models—systems that process both text and images—claim to reason by generating intermediate visual artifacts: sketches, annotations, step-by-step diagrams. But do they actually rely on these visuals, or are they just window dressing for text-based reasoning? This paper challenges that assumption directly and finds a sobering answer: models struggle to faithfully render what they intend, and visual states are far less trusted than their behavior suggests.

The authors introduce See2Think, a diagnostic framework designed to expose whether models genuinely depend on visual reasoning steps. The framework has two parts. First, See2ThinkBench is a benchmark of 1,200 open-ended problems across 12 task categories—spanning 2D structured diagrams, 3D scenes, and real-world reasoning—all designed to require visual understanding. Second, Visual Action-of-Thought (VAoT) is an evaluation protocol that doesn't just check final answers. Instead, it logs every step: what the model says it will do (textual thoughts), what operations it performs (visual actions), what gets rendered on screen (the actual image), and how the model reasons afterward.

What the Paper Actually Found

Testing both proprietary and open-source models, the authors uncovered three key failures:

1. No universal setting wins. Models perform differently depending on whether they reason with all visual steps visible, with only text summaries, or in other configurations. There's no single best approach—it's task-dependent.

2. Rendering is the real bottleneck. Models often select the right visual operations—picking sensible next steps—but then fail to execute them faithfully. A model might decide to "annotate the triangle in red," but the rendered output is garbled or inaccurate. This gap between intention and execution breaks downstream reasoning.

3. Models show false feedback uptake. When given visual feedback (corrupted or not), models appear to incorporate it—they adjust their next steps in response. But this correlation is weak. When researchers intentionally feed incorrect visual information and measure accuracy drop, they see losses of 10+ percentage points, proving models do depend on visual states. Yet models often claim to use visual feedback when they don't truly need it, suggesting shallow integration.

Why This Matters

The findings deflate a common assumption in multimodal AI research: that intermediate visual reasoning is a solved problem or that models have truly learned to "see and think." Instead, the paper reveals that visual reasoning is fragile, inconsistent across settings, and bottlenecked by rendering fidelity rather than reasoning logic. A model might generate sensible descriptions of what it should draw, but fail at the actual drawing step—and then fail again when trying to interpret the flawed output.

For practitioners, this suggests that simply prompting models to "think visually" isn't enough; the infrastructure for generating and rendering intermediate states matters enormously. For researchers, it highlights that benchmarks checking only final answers miss critical failure modes. The paper's diagnostic approach—tracking thoughts, actions, and rendered states separately—provides a template for probing whether future models truly see or merely simulate seeing.