AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

The Problem: Audio-Visual Editing Falls Through the Cracks

Most video editing benchmarks treat sound and picture separately, if they evaluate audio at all. Real-world videos don't work that way—change the dialogue and you may need to adjust lip-sync; add background music and the visual mood shifts. Existing systems excel at pure visual edits on mute or pure audio tweaks in isolation, but struggle when instructions span both modalities and require one to respect changes made to the other. There's no standard way to measure whether an edit works together.

The authors introduce AVE-Compass, a benchmark designed to close this gap. It includes 145 real videos, 196 audio-visual editing instructions, and 2,688 checklist items for human evaluators to verify. The benchmark measures four dimensions: whether the model followed instructions, whether it preserved untouched content, whether the result feels natural, and whether the edit achieves the intended effect. Assessment combines human judges (using multimodal language models to read checklists), dedicated realism rubrics, and automated metrics that check cross-modal alignment, video quality, and audio quality separately.

Results: Models Struggle at the Junction

When the authors tested state-of-the-art models, they found that current systems perform well on single-modality tasks but fail at cross-modal consistency. For example, a model might change dialogue without adjusting visuals to match, or add music without accounting for how it clashes with existing sound. Even when models do edit both modalities, they often corrupt non-target content—the "Fidelity Preserving" dimension reveals this weakness clearly.

The Solution: Break It Down, Then Iterate

To improve results, the authors propose AVE-Agent, a modular framework that treats a complex audio-visual instruction as a sequence of dependent subtasks. Rather than trying to edit everything at once, the agent decomposes the task (e.g., "replace the background music and adjust the dialogue volume"), executes each step, and uses self-reflection plus evaluator feedback to refine the outcome iteratively. This approach strengthens instruction execution, content preservation, and audio-visual alignment while keeping the final product perceptually competitive.

Why This Matters

Audio-visual editing is common in real-world workflows—podcasts, films, social media—yet it's been invisible in academic benchmarks. AVE-Compass makes that gap visible and measurable. By introducing both a rigorous evaluation framework and a working solution that outperforms end-to-end baselines, the paper signals that cross-modal consistency is a tractable research problem, not an afterthought. The checklist-based evaluation with MLLM judges is also a practical contribution; it scales human judgment without requiring thousands of manual annotations per video. For practitioners building editing tools, this work offers concrete evidence of where current systems fail and a proof-of-concept for how to fix them through decomposition and feedback loops.