MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation

The Problem: No Good Way to Measure Multi-Reference Video Generation

Generating videos and audio from multiple reference sources at once is hard. A model might need to take a photo of a person, a sound clip, and a text instruction ("make them dance to this music in a kitchen"), then produce coherent audio-video output that respects all three inputs. But today's benchmarks don't actually test this. They measure single-reference preservation, text-to-video in isolation, or audio-video sync separately. Nobody has built a benchmark that checks whether a model can faithfully pull multiple references together while keeping the audio and video synchronized and following instructions—what the authors call the MR2AV setting.

The authors introduce MultiRef-Compass, a benchmark with 350 hand-curated samples and a structured evaluation framework to fill this gap.

How It Works

The benchmark has three layers: data, evaluation protocol, and judging.

Data construction uses a "scalable and controllable asset-composition pipeline." In plain terms: the authors assemble short videos from reusable components (a person, an object, a background scene) with known relationships, then add text instructions and paired audio. This gives them three test categories: multi-view subject preservation (same person from different angles), multi-entity binding (combining separate characters or objects into one scene), and human-object-scene composition (a person interacting with objects in an environment).

Evaluation has four dimensions spanning 14 metrics:

• Basic Quality: standard metrics like resolution, flicker, and temporal coherence.
• Reference Consistency: does the generated output look like the reference sources? Measured via embedding similarity and CLIP alignment.
• Audio-Visual Consistency: are the lip movements, actions, and sounds synchronized?
• Instruction Following: did the model do what the text asked?

Rather than relying on a single automatic metric (which can miss nuance), the authors combine automated measurements with an MLLM-as-a-Judge framework—a large multimodal language model that watches the video and answers detailed questions about each dimension. To make this auditable and scalable, they add a "rejudging" step where the model's assessment is reviewed, improving consistency.

Why It Matters

Testing eight existing MR2AV systems revealed that none excel across all dimensions. Most preserve individual references reasonably well but struggle to bind multiple entities correctly or maintain audio-visual sync. This finding is concrete evidence that the problem is harder than prior benchmarks suggest.

For researchers, MultiRef-Compass provides a shared standard—something the field lacked. The 350 curated samples are small enough to evaluate quickly but large enough to reveal real weaknesses. The four-dimensional protocol avoids the false comfort of a single score; it exposes which aspects of the generation pipeline break down.

By grounding evaluation in interpretable dimensions and combining human-like judgment with scalable automation, the benchmark positions itself as infrastructure for the next generation of MR2AV models. The substantial gaps the authors found suggest the field is still in its infancy.