VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

The Problem: Metrics Miss the Big Picture

Generating videos and audio that match a text description is hard. Currently, researchers train these models using separate quality metrics—one for audio clarity, one for visual sharpness, one for how well the sounds sync with the video. The catch: optimizing for these individual metrics doesn't guarantee humans will like the result. A video might score high on all three dimensions yet feel incoherent to a viewer—the speech doesn't match the lip movements, the background music clashes with the scene, or the content doesn't match what was requested. Metrics reward gaming over genuine coherence.

This paper argues that human preference depends on holistic semantic and temporal alignment across text, video, and audio together—something that individual quality scores completely miss.

The Solution: Learning From Human Judgments

The authors built VA-Judger, a reward model that learns what humans actually prefer by studying their choices. Their approach unfolds in three stages:

Stage 1: Building a preference dataset. The team created VAPref-10K, containing 9,000 text prompts with 10,300 paired video-audio comparisons rated by humans. This grounds the reward model in real human judgment rather than handcrafted metrics.

Stage 2: Training with a chain-of-thought strategy. Rather than jumping straight to ranking hard cases, VA-Judger first learns from easy comparisons—pairs where one output is clearly better. This establishes structured reasoning. Then, for trickier near-equal pairs, the model generates preference explanations and only keeps explanations that match human annotations (rejection sampling). This forces the model to learn why humans prefer one output, not just memorize rankings.

Stage 3: Decomposing feedback for denser rewards. Once the model understands overall human preference, it breaks that preference back down into individual dimensions (audio quality, visual fidelity, sync, etc.). This gives reinforcement learning more granular training signals than a single binary "which is better?" label.

Why It Matters

Experiments on VA-Judger-Bench—a new benchmark with both in-domain (similar to training data) and out-of-domain (unseen models) comparisons—show VA-Judger predicts human preferences far better than metric baselines. More importantly, when used as a reward signal to post-train a video-audio generation model, it yields measurably better outputs that humans rate higher.

This sidesteps reward hacking: the model can't game individual metrics because it's being optimized against genuine human preference. The chain-of-thought design keeps explanations honest by verifying them against annotations. The three-stage training avoids overfitting to easy cases and gradually tackles harder judgments.

The work addresses a real gap in multimodal generation: existing methods treat vision and audio as separate problems, but humans evaluate them as a unified experience. VA-Judger brings that holistic perspective into the reward signal, making post-training more likely to produce content that feels right, not just content that scores well on paper.