IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products

## The Gap Between What Models See and What They Should Extract

Industrial products—valves, circuit breakers, transformers—come with dense technical specifications scattered across multiple images: specification tables, nameplates, technical drawings. The authors ask a simple but crucial question: can multimodal large language models (MLLMs) actually recover all these scattered specifications? The answer is sobering. Even the best models find only about half of the product-level attributes, despite claiming high precision on individual images.

This matters because procurement, supply chain compatibility, and safety depend on complete, accurate specifications. A missed attribute isn't just a small error—it can break a supply chain or create a safety risk.

## How the Benchmark Works and What It Reveals

The authors built IndustryBench-MIPU as the first large-scale benchmark for this task. They assembled 4,559 industrial products across 27,652 images with 103,703 property-value pair annotations, covering 18 industrial categories. The data came through multi-model consensus and three-tier quality checks to ensure reliability.

The core task is structured attribute extraction: given multiple images of a product, extract all the property-value pairs that define it. This isn't just OCR—it requires text recognition on specification tables, visual reasoning over technical drawings, understanding of domain-specific terminology, and the ability to integrate evidence across images. A single image might show the front panel; another shows the datasheet; a third shows internal components. Models must synthesize across all of them.

They evaluated nine MLLMs (unspecified in the abstract, but representing the landscape of current multimodal systems) in two settings: single-image extraction and product-level multi-image extraction.

## The Core Finding: Multi-Image Integration Fails

The results expose a critical bottleneck. Models achieve high precision (86–94%)—when they find an attribute, they usually get it right. But they are incomplete. The best-performing model recovers only 49.9% of product-level attributes. Moving from single-image to multi-image extraction costs 15–34 percentage points of recall. This gap is enormous. Models either fail to recognize that an attribute appears across images or struggle to integrate scattered evidence into a coherent product specification.

In other words, the problem is not hallucination or false positives; it's systematic undercounting. Attributes exist in the images, but the models miss them—especially when they require cross-image reasoning.

This has immediate practical implications. In supply chain and procurement workflows, missing attributes are dangerous. A 50% completion rate is unusable, regardless of precision. The benchmark quantifies exactly where current MLLMs fail: not at reading individual images, but at the multi-image assembly task that industrial product understanding actually demands.

The authors released the dataset and code, positioning this as an open challenge for the community to improve multi-image reasoning in domain-specific, high-stakes contexts.