InSight-doc: Agentic Visual Perception for Long-Document Understanding

The Problem: Long Documents Cost Too Much to Process

Long-document understanding—extracting answers from multi-page PDFs, reports, or scanned materials—hits a hard limit: feeding all pages at full resolution through a vision-language model is computationally expensive and often makes the model "hallucinate" (confidently state wrong information). The authors identify a simpler culprit than most solutions: the model wastes compute by processing entire pages uniformly when only a few regions actually matter for the answer.

InSight-doc flips this around. Instead of upfront full-resolution processing, it starts with a coarse overview and learns to zoom strategically—only examining high-resolution patches where evidence likely exists. This adaptive strategy cuts inference time by 41–68% and reduces hallucination by over 40% on long documents, while gaining 4.3–16.4 accuracy points on standard benchmarks compared to non-adaptive baselines.

How It Works: Agent-Guided Selective Zoom

The core idea is treating resolution as a "reasoning-time resource" the model allocates on demand. InSight-doc operates in stages:

1. Start low. The model receives a low-resolution overview of all pages, getting global context cheaply.
2. Decide where to look. Acting like an agent, it selects specific regions that might contain relevant evidence.
3. Zoom and refine. It re-examines chosen regions at high resolution and updates its reasoning.
4. Iterate or conclude. Repeat until confident enough to answer, or stop if diminishing returns.

Critically, this doesn't require an external retriever (a separate system that pre-identifies relevant sections). The model learns end-to-end which regions to prioritize.

Training the Agent: Data and Learning

To teach the model this zoom-and-reason behavior, the authors built two datasets:

• 17.9K SFT examples (supervised fine-tuning): high-quality trajectories showing where to zoom and why, labeled at the region level. Each example traces a path from low-res overview through selective high-res glimpses to the final answer.
• 19.2K hard RL examples (reinforcement learning): tougher cases where naive strategies fail, used to refine the zoom policy through trial-and-error rewards.

They combine both approaches—first SFT to bootstrap the agent, then RL to optimize for real-world accuracy and efficiency trade-offs.

The resulting 8B-parameter model shows strong gains: on document VQA benchmarks (visual question answering over pages), it outperforms the non-adaptive 8B baseline by 4–16 points. On longer documents, the advantage grows because the model avoids wasteful processing of irrelevant pages.

Why This Matters

Document understanding powers practical applications—contract review, research paper analysis, invoice processing. Most solutions either sacrifice speed (full-resolution processing) or accuracy (heavy summarization). InSight-doc's selective zoom approach offers a genuine trade-off lever: use less compute, hallucinate less, and answer more accurately by learning what matters.

The release of code, datasets, and the trained model lowers the barrier for others to build on this approach, potentially opening a research direction around adaptive visual perception in multimodal AI.