OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

The Problem: Modality Conflicts in Token Compression

Omnimodal large language models (OmniLLMs) process audio, video, and text simultaneously, but they generate enormous numbers of tokens—far more than efficient inference allows. Existing compression methods make a critical assumption: whichever tokens matter most in video also matter most in audio, so one modality can guide pruning decisions for the other. OmniScope's core finding demolishes this assumption. For the same query, audio and video often peak in relevance at completely different moments. When one modality dominates the pruning decision, the other loses answer-critical information, especially under aggressive compression ratios.

How OmniScope Works

The method abandons the idea of shared salience estimates and instead treats audio and video as independent. The query—the user's question—serves as the only shared semantic reference point across both modalities.

Modality-specific budgets: OmniScope allocates separate token retention targets for audio and video based on their individual relevance to the query, not a global compression ratio. This allows audio to keep tokens densely in seconds where speech carries crucial information, while video preserves tokens in scenes where visual details matter.

Visual pruning: For video, the framework uses an "anchor-delta strategy" that preserves two types of information simultaneously. Anchor tokens capture the global visual context (what's in the frame), while delta tokens track temporal changes (motion and transitions). This dual approach prevents both the loss of static scene understanding and the collapse of temporal coherence.

Audio merging: Rather than pruning, audio tokens are merged within each one-second window. This reduces redundancy—consecutive audio frames often contain near-identical acoustic information—while keeping the temporal structure intact. No information is discarded, only compressed.

Results and Impact

Testing across four audio-video benchmarks (likely including tasks requiring understanding of speech, visual content, or both), OmniScope outperformed baselines at every compression level on Qwen2.5-Omni models at two different scales. At extreme compression (25% of original tokens retained), the method achieved:

• 3.53x speedup during prefill (the initial pass where the model processes all input tokens)
• 15%+ GPU memory reduction
• Only 0.35-point average accuracy drop

The practical significance is substantial: omnimodal inference becomes feasible on resource-constrained hardware without catastrophic quality loss.

Why This Matters

The core insight—that modality-decoupling is crucial for compression—challenges a widespread design pattern in multimodal AI. By treating audio and video salience independently while sharing only the query semantics, OmniScope enables genuinely efficient omnimodal inference. The method is also training-free, meaning it can be applied to existing pretrained models without retraining. For practitioners deploying OmniLLMs at scale, this translates to real cost and latency gains while maintaining quality, opening doors for on-device omnimodal AI applications where inference speed and memory are hard constraints.