On Locality and Length Generalization in Visual Reasoning

Local Vision Works Better Than Global—When Tasks Demand It

Vision models trained to process images all at once (globally) tend to exploit shortcuts and fail when task complexity increases. Models that instead process images sequentially through small, local windows—mimicking how human eyes foveate—generalize far better to harder versions of the same task. This finding challenges the dominant paradigm in computer vision and suggests locality isn't just biologically plausible; it's computationally useful.

The Problem: Shortcuts Over Composition

The authors tested vision models on tasks requiring them to aggregate information scattered across an image—think counting objects or tracking state changes in a visual scene. Models trained on simple versions (short sequences, low complexity) should learn compositional strategies that work on harder versions. Instead, they discovered models learn lazy heuristics tied to task length. A model trained to count objects in 5-object scenes learns to exploit global patterns specific to that length rather than actually counting, so it fails at 10 objects.

This mirrors a known weakness in language models: transformers trained on sequences of length n often fail catastrophically on length n+1, even for simple tasks like copying or arithmetic. The authors bring this lens to vision.

The Solution: Recurrent Local Policies

The fix is a recurrent vision policy that processes the image step by step, attending only to a small local window at each step. At each timestep, the model:

• Views a limited spatial region (strictly local attention)
• Updates an internal hidden state
• Moves to the next region

This forces the model to track state compositionally across steps rather than memorize global patterns. Because the model never sees the entire image at once, it cannot exploit length-dependent shortcuts. It must learn an algorithm that generalizes.

Why This Matters

Robustness through constraint. Humans see foveated snapshots because our eyes are high-resolution in the center only. This biological constraint turns out to be a feature, not a bug—it pushes models toward generalizable solutions. The paper argues that what seems like a limitation is actually an inductive bias that prevents overfitting to spurious statistics.

Compositional generalization. Many vision and reasoning tasks are compositional: the solution to a 3-object problem should scale to 10 objects. Locality achieves this better than global models, opening doors for more reliable visual reasoning systems.

Bridging vision and language. This work connects insights from language model scaling research back to vision, showing that length generalization failures aren't language-specific—they're a broader failure of models to learn compositional structure when global access allows shortcuts.

The experiments use simple synthetic visual reasoning tasks (the paper doesn't detail datasets in the abstract, but the design clearly targets controlled, measurable generalization). The recurrent local approach consistently outperforms global baselines on out-of-distribution test sets.

This paper reframes an old idea—local receptive fields—as a solution to a modern problem: preventing neural networks from learning brittle, length-dependent heuristics.