Evidence-Backed Video Question Answering

The Problem: Video AI Can Answer Questions, But Nobody Knows Why

Current video understanding systems—called Video Large Language Models—can answer questions about video content, yet they offer no proof. Ask "what happens to the cup?" and you get an answer, but no visual evidence showing where the cup is, when it moves, or how it changes shape. Existing attempts at transparency use vague text explanations or a few scattered boxes, which fail entirely when objects partially hide each other, bend, or deform.

This paper argues that real explainability requires precise visual grounding: showing exactly which video frames matter and which pixels belong to relevant objects at each moment.

The Solution: Spatio-Temporal Evidence

The authors propose E-VQA (Evidence-Backed Video QA), where models must produce two outputs for each question: a semantic answer plus visual proof. The proof has two parts:

1. Temporal segments: which specific time intervals in the video are relevant
2. Tracked object segmentation masklets: pixel-level masks showing where objects appear across frames, with masks that follow the same object continuously (tracking)

To make this work, they built ST-Evidence, the first human-annotated benchmark with this dual requirement. They evaluated existing state-of-the-art models and found a striking disconnect: models that answer questions correctly often fail to point at the right pixels—showing that scaling model size alone doesn't fix visual grounding.

The Data Fix: Scaling Grounding via Automation

Rather than manually annotate thousands of examples, the authors created ST-Evidence-Instruct, a 160k-video dataset built through automated pipelines. These pipelines generate spatio-temporal grounding automatically, then use human feedback to verify quality.

Fine-tuning 7B-parameter Video LLMs on this synthetic-but-verified data yielded large gains over baselines: +27.2 points on temporal accuracy (t-mean metric) and +13.8 points on object segmentation quality (J&F metric). Crucially, these gains matched or exceeded models that were simply trained on larger datasets with no grounding supervision.

Why This Matters

Video AI is moving into high-stakes domains—autonomous vehicles, medical imaging, surveillance—where "trust me, it's right" is unacceptable. This work forces a reckoning: answering a question and understanding the visual content are not the same thing. The gap persists even in state-of-the-art models, and brute-force scaling doesn't close it.

The paper's practical contribution is concrete: a reusable dataset and a training recipe that makes Video LLMs both accurate and transparent. By explicitly demanding pixel-level, temporally-aware evidence, the work moves explainability from optional post-hoc rationalization to built-in accountability.