SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

The Problem with Current Benchmarks

Existing visual document understanding benchmarks like DocVQA and ChartQA are too tangled. When a model fails, you can't tell why — is it struggling with document length, complex layouts, mixed content types, or hard questions? Real documents combine all these factors at once, making it impossible to diagnose weaknesses. The authors' central claim is that current benchmarks have blind spots, and they've built a synthetic benchmark that isolates each factor to expose what models actually can't do.

How SynthDocBench Works

The benchmark generates documents synthetically using an LLM pipeline, giving the researchers complete control. Instead of using real-world documents where variables are hopelessly entangled, they construct documents across six layout archetypes — basic templates that vary independently. They systematically dial up document length, change layout structure, mix modality composition (text, charts, tables), and adjust question difficulty.

The key methodological move is a 40 percent random override. After generating a document, researchers randomly replace content in 40 percent of cases to prevent models from learning spurious shortcuts. This mimics the messiness of real documents without sacrificing control.

Documents in SynthDocBench are substantially longer and structurally more diverse than existing benchmarks, pushing models into unfamiliar territory.

Three Critical Failure Modes

When the authors evaluated seven frontier vision-language models (GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, LLaVA-NeXT, Qwen-VL, Phi-4V, and others), three failure patterns emerged that existing benchmarks completely miss:

Sharp degradation with length. Model accuracy drops steeply as documents grow longer — a real problem that short benchmarks never reveal.

Positional sensitivity. Five of six models perform worst on questions about the middle third of documents. Questions about the first third and last third are easier. This is a striking systematic bias that suggests models aren't processing documents holistically.

Early-to-Late trend. Five of six models show steeper performance declines for later parts of documents (up to 8.3 percentage points), indicating they may attend more to the beginning and degrade as context accumulates.

Most damning: chart comprehension breaks down entirely in long-document settings, even for models that handle charts well in isolation.

Why This Matters

These results suggest frontier VLMs aren't achieving genuine long-context document understanding — they're overfitting to the specific artifacts and biases in existing benchmarks. A model that performs well on DocVQA may not actually understand how to reason across long, complex documents because those benchmarks don't test the right hard cases.

SynthDocBench enables researchers to isolate failure modes and build toward actual robustness rather than benchmark-gaming. By controlling individual factors, the benchmark also serves as a diagnostic tool: developers can now ask "does my model fail because of length, or layout, or something else?" and get a clear answer.