OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

The Problem: You Can't Benchmark a Real-Time Assistant with a Static Dataset

Video understanding models have improved dramatically, but there's a blind spot: evaluating them as interactive assistants. Traditional benchmarks measure passive comprehension—"describe this video." But a real assistant must do something harder: watch continuously, understand what the user wants, predict what they'll do next, and guide them step-by-step toward a goal. Every response the model gives changes what the user does afterward, creating branching paths that no fixed dataset can capture. Existing Omni-LLMs (models that handle text, images, video, and audio together) have no rigorous way to measure interactive performance.

The authors' solution is OmniAssistBench, a dataset and evaluation framework built specifically for multi-turn, goal-driven video interactions. The key insight: rather than trying to capture every possible path through an interaction, they lock models into following one correct path by providing predefined "priors"—visual landmarks or state information extracted from the source video. This makes evaluation deterministic and reproducible, while still forcing the model to provide moment-by-moment guidance that adapts to user actions.

How They Built It (and Why It's Hard)

Genuinely interactive video data doesn't exist at scale—nobody has thousands of recorded sessions where users follow instructions while being corrected in real time. So the authors reverse-engineered existing Internet videos. They watched clips, inferred plausible user goals ("learn to fold origami," "repair a bicycle"), then worked backward to identify what an assistant should say at each step. They segmented videos into multi-turn exchanges, creating scenarios where a user takes an action, the model responds with guidance, and that response should steer the user toward the next correct frame.

This labor-intensive process consumed over 1,000 expert person-hours. The rigor matters: it ensures that "correct" responses are unambiguous (tied to specific video frames and predefined goals), not subjective.

What the Results Reveal

The leaderboard is sobering. Gemini-3-Pro (proprietary) scores 66.4 out of 100; Qwen3-Omni-Instruct (open-source) reaches 51.2. These are not failing grades, but the gap to reliability is wide. The breakdown of failures is telling:

• Visual prompts fail: Models miss hand gestures and other visual cues meant to signal intent.
• Context degradation: Over multiple turns, models forget earlier parts of the conversation or lose track of the user's progress.
• Timing errors: Models respond too early or too late, unable to wait for a target event (e.g., "tell me when the user's hand reaches the bucket").

In other words, current Omni-LLMs can parse individual frames and understand language, but they struggle with the sequential, context-aware reasoning that real assistance demands.

Why This Matters

As voice assistants and AR glasses proliferate, there's genuine demand for models that can watch a live scene and guide users in real time. This benchmark fills a concrete gap: it's the first rigorous evaluation framework for this use case. The authors have shown not just where models fail, but how they fail—providing a roadmap for improvement. Without OmniAssistBench, progress on interactive video understanding would be invisible.