DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

The Challenge: Real-World Data is Messy

When organizations ask data questions, the answer often lives nowhere in a single place. Files scatter across CSV tables, JSON documents, PDFs, videos, and databases. Current benchmarks for AI data agents sidestep this reality—they test structured queries in isolation, or retrieval tasks alone, or open-ended analysis without checkable answers. DataSpace fixes that gap by forcing agents to hunt through genuinely heterogeneous workspaces and produce verifiable tabular results that can be scored objectively.

The authors created a benchmark with 410 tasks spanning six languages, grounded in 7,439 real artifacts (15 GB total) across CSV, JSON, SQLite, Markdown, PDF, and video. Each task gives an agent only a question and a workspace folder; it must return a complete, correct table. This mirrors what a real analytics system actually needs to do.

How They Built It—And How Agents Are Scored

DataSpace-Builder handles the messy engineering. It converts artifacts across formats (cross-language transformation), samples data smartly to avoid overwhelming agents while respecting constraints, routes queries to the right modality (video? PDF? database?), and renders each artifact for the agent to consume. Eleven domain experts then reviewed and repaired tasks to ensure they're sensible and solvable. The stakes were high: this became the official benchmark for KDD Cup 2026.

Scoring is deterministic but practical. Since tables can be written in different column orders or with minor formatting differences, the evaluator aligns columns without caring about header names, normalizes types and precision, and compares rows while tolerating row order changes. This avoids penalizing agents for cosmetic differences.

The Results—And Why This Matters

The authors tested six recent frontier multimodal models (like Claude, GPT-4V, Gemini) with five different agent frameworks (the software that orchestrates how the model interacts with the workspace). The best accuracy hit 66.34%—still far from solved. More troubling: swapping the agent harness alone, without changing the backbone model, caused a 15.36-point accuracy swing. This means the way you ask a model to work through a problem matters as much as the model itself.

Two failure modes stand out. First, integrating evidence across modalities (joining a PDF table with a CSV, or cross-referencing video timestamps with database records) consistently tanks performance on all six models. Second, joins themselves—even within structured data—are a persistent weak point. These aren't minor glitches; they're architectural gaps in how current agents understand multi-source analytics.

DataSpace is unsaturated by design. With two-thirds accuracy at best and clear pain points in multimodal retrieval and relational operations, the benchmark identifies concrete directions for improvement. For researchers building data agents, this is both a rigorous evaluation bar and a roadmap: improve cross-modality reasoning, strengthen join logic, and invest in reliable evidence synthesis across heterogeneous sources.