CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

The Gap in Multimodal Learning Nobody's Measuring Yet

Current benchmarks test whether AI models can understand text and images they've seen during training. But real work—reading a research paper with figures, analyzing financial reports with charts, navigating maps—requires learning new information from multimodal context in the moment. The authors find that existing evaluations ignore this critical capability, and they've built CLBench-V to expose how badly models fail at it.

Across six recent multimodal models tested on 3,443 examples, the best score is just 28.5%—a clear signal that multimodal context learning is genuinely unsolved, not a minor gap.

What the Benchmark Actually Tests

CLBench-V organizes multimodal context learning into three dimensions that pinpoint where models break down:

Context grounding: Can the model identify which parts of the image or document are relevant? A model might see a scientific figure but struggle to extract the specific data point needed to answer a question.

New information application: Does the model actually use novel facts it found in the context to solve the task? This tests whether it can move beyond pattern matching to genuine reasoning with fresh information.

New knowledge learning: Can it acquire and retain genuinely new conceptual understanding—facts that weren't in its training data?

This three-way split matters because a model might ground context perfectly but fail to apply it, or apply it once but not retain it. Traditional benchmarks lump all these together.

The Datasets and Construction

The benchmark spans five domains: science (figures in papers), finance (charts in reports), long-document understanding (dense text + tables), spatial reasoning (maps and scenes), and web-based visual QA (screenshots). Some tasks come from converted public benchmarks; others were newly built using automated construction and filtering to reduce the manual labor of creating domain-specific datasets.

All told, 3,443 instances across these domains, evaluated on models including InternVL3.5-30B-A3B and Qwen3.5-Plus.

Who Wins, and Why It Matters

Interestingly, no single model dominates across all three dimensions. InternVL3.5-30B-A3B leads on context grounding and new knowledge learning, while Qwen3.5-Plus performs best on new information application. This fragmentation suggests different architectural or training choices help with different aspects of the problem—insights that guide future development.

The authors also analyze failure modes by context length and image count, revealing which problem types remain hardest. At 28.5% best performance, these aren't edge cases—they're core failures that will matter as multimodal models move from research into real-world deployment.

The takeaway: If your job involves learning from mixed text-and-image documents on the fly, current models are far from reliable. CLBench-V gives researchers a structured way to measure and fix that gap.