Lode

Stand on the shoulders of giants.

Open the curator →
Source
arXiv
Published
Runtime
0:00
Snippets
5

A conversation between

Matteo Farina , Vishaal Udandarao , Thao Nguyen , Selim Kuzucu , Maximilian Böther , Andreas Hochlehnert , Adhiraj Ghosh , Marianna Nezhurina , Karsten Roth , Joschka Struber , Yuhui Zhang , Sebastian Dziadzio , Elaine Sui , Soumya Jahagirdar , Dhruba Ghosh , Hasan Hammoud , Thomas De Min , Simone Caldarella , Jehanzeb Mirza , Sedrick Keh , Mehdi Cherti , Hilde Kuehne , Bernt Schiele , Serena Yeung-Levy , Muhammad Ferjad Naeem , Federico Tombari , Ana Klimovic , Elisa Ricci , Matthias Bethge , Sewoong Oh , Ameya Prabhu , Alessio Tonioni , Jenia Jitsev , Massimiliano Mancini , Ludwig Schmidt , Nikhil Parthasarathy

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Source · arxiv.org/abs/2606.28551 ↗

0:00 0:00

§02

Snippets

  1. DataComp-VLM is a benchmark with 160 datasets (6T multimodal tokens) that lets researchers test filtering, mixing, formatting, and sampling strategies across controlled model and compute scales.

    This is the first systematic way to evaluate data curation for VLMs, moving past ad-hoc dataset choices to reproducible, comparable experiments.

  2. Data mixing strategies outperform filtering; instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger model scales.

    This flips conventional wisdom and suggests the composition of training data matters more than purity, reshaping how practitioners should build datasets.

  3. DCVLM-Baseline achieves 63.6% on a 33-task core suite with an 8B model and 200B tokens, a +5.4pp gain over FineVision (prior state-of-the-art open dataset).

    Demonstrates that systematic data curation can meaningfully close the gap with proprietary datasets, validating open-source competitiveness.

  4. DCVLM evaluation spans up to 52 downstream benchmarks across 9 domains, providing a comprehensive and consistent evaluation suite for all submissions.

    Standardized evaluation enables fair comparison and builds confidence that observed improvements generalize beyond cherry-picked tasks.

  5. DCVLM integrates four data types (image-captions, multimodal documents, text-only, instruction data) and allows testing custom mixing and formatting strategies.

    Enables researchers to systematically explore how different modality combinations and data formats interact, rather than guessing.

§03

Synthesis

DataComp-VLM: A Benchmark for Better Vision-Language Model Training Data

The central finding is counterintuitive: when training vision-language models, how you mix different data sources matters far more than filtering out low-quality examples. A new benchmark called DataComp-VLM (DCVLM) demonstrates this across systematic experiments, and the resulting best-practice dataset improves state-of-the-art open-source VLM performance by 5.4 percentage points.

The Problem and Benchmark Design

Vision-language models—systems that process both images and text—need enormous training datasets to work well. Until now, there was no standardized way to test which data curation strategies actually help. Teams built datasets ad hoc, making it impossible to isolate what works.

DCVLM changes this by providing a controlled experimental framework. The authors assembled 160 datasets totaling 6 trillion multimodal tokens across four data types: image-caption pairs (like LAION), multimodal documents (images mixed with text), text-only data, and instruction-tuning examples (prompts with answers). Researchers can now test different curation strategies—filtering, mixing ratios, data formatting, sampling methods—on models ranging from 1 billion to 8 billion parameters, with training budgets from 6.25 billion to 200 billion tokens. Performance is then measured on up to 52 downstream benchmarks spanning 9 domains (vision, language, reasoning, etc.).

Key Finding: Mixing Beats Filtering

The core surprise: filtering (removing "bad" data) doesn't significantly boost performance. Instead, the composition of the mixture matters. Instruction-heavy datasets—those with question-answer pairs and explicit tasks—scale better than caption-heavy ones, especially at larger training budgets. This pattern held consistently across all model sizes tested.

The authors' best-practice dataset, DCVLM-Baseline, incorporates these insights. When training an 8 billion parameter model on 200 billion tokens, it reaches 63.6% accuracy on a core suite of 33 tasks. This beats FineVision, the previous best open-source VLM training dataset, by 5.4 percentage points—a meaningful gain given the scale.

Why It Matters

This work addresses a bottleneck in open-source VLM development. Proprietary models (GPT-4V, Claude) benefit from massive curated datasets, but open models have lagged partly because the community lacked guidance on data curation. DCVLM levels the playing field by making both the benchmark and the resulting best-practice dataset publicly available.

The finding that mixing strategies outweigh filtering is also practically useful: it suggests that practitioners should focus effort on thoughtfully combining diverse data sources rather than spending cycles on quality filtering, which is often subjective and expensive. The instruction-heavy insight points toward a concrete recipe for future work.

By releasing all artifacts alongside the paper, the authors create a reusable benchmark for the field—similar in spirit to their earlier DataComp project for text-image models, but tailored to the multimodal and instruction-tuning challenges of modern VLMs.

Mine your own.

Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.

Open the curator