DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation

The Core Problem and Solution

Visual document retrieval (VDR)—finding relevant documents by their images when queried with text or images—currently relies on enormous models with billions of parameters. These systems are expensive to run and slow to index at scale. The authors show that you can shrink an 8-billion-parameter teacher model down to a compact 524-million-parameter student that retrieves just as well, without needing expensive supervision like relevance labels or negative examples during training.

How Distillation Works Here

Traditional knowledge distillation copies what a large model learns, but DistilVDR takes a simpler route: it trains a student to match the embedding space of a frozen teacher. Instead of mimicking complex behaviors, the student learns to produce vectors (numerical representations of documents and queries) that align with the teacher's vectors using a cosine similarity loss. All the supervision—whether a document is relevant to a query—comes pre-built into the teacher's embeddings. The student doesn't need to re-learn relevance from scratch.

The clever part is the asymmetry. Text queries are short and simple; document images are rich and high-dimensional. The authors give their student a lopsided architecture: the query encoder has only 70 million parameters, while the document encoder absorbs most of the 524M capacity. This mismatch mirrors how VDR actually works and avoids wasting compute on the query side.

Why This Matters

The results validate the approach. On the ViDoRe benchmark (a standard VDR evaluation spanning three versions), DistilVDR-HiRes achieves 61.74 NDCG@5, which is 86.9% of the 8B teacher's performance—a strong recovery for a 15× smaller model. More importantly, it outperforms every reproduced baseline under 1 billion parameters, especially on the high-resolution-sensitive v3 benchmark where pixel detail matters.

For deployment, the gains are substantial: the index is 15.6× smaller than the next-best sub-1B multi-vector baseline, and indexing a million documents runs an order of magnitude faster. A variant called DistilVDR-Fast trades a bit of accuracy (59.98 NDCG@5) for a 3× smaller visual-token budget, useful when latency is the priority.

By training both variants with identical encoders and only varying the visual-tile budget in the document encoder, the authors show that the distillation technique is robust; you can dial up or down visual resolution depending on your constraints. The method needs no contrastive learning or hard-negative mining—just pointwise alignment to the teacher—which simplifies training and reduces memory overhead.

This work opens a practical path to deploying VDR at scale without running billion-parameter inference on every query.