TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

The Challenge: When, Not Just What

Video multimodal language models (MLLMs) can tell you what happens in a video, but they struggle to pinpoint when it happens. TimeLens2 tackles video temporal grounding—the task of finding all the time intervals in a video that support an answer to a query, even when there are multiple disconnected moments. The catch: those intervals vary in number and location, across videos of different lengths, question types, and viewing angles. Existing methods fail because they treat this as a set-finding problem using tools designed for single-interval or ordered-sequence tasks.

The authors' key insight: temporal evidence is fundamentally a set of intervals, not a single span or sequence. Training and optimization should respect that structure throughout, rather than patching it on at the end.

Method: From Annotation to Reward

Building reliable multi-span labels. The authors created TimeLens2-93K, a dataset with multiple evidence intervals per query. Rather than relying on fragile one-pass human annotation of long videos, they derive proposals from video captions, then validate them through independent localization by different annotators, cross-agent consensus checking, semantic verification (confirming the interval actually supports the answer), and boundary refinement.

Wasserstein-based reward for set matching. This is the technical innovation. Standard RL rewards for temporal grounding either ignore non-overlapping predictions entirely or require brittle segment-by-segment matching that breaks when predictions and labels have different cardinalities (different numbers of intervals).

TimeLens2 uses the temporal Wasserstein distance (W₁), which treats each merged set of intervals as a uniform distribution over time and computes the exact one-dimensional distance between them. This provides dense feedback even when predictions and labels don't match interval-for-interval. A complementary temporal IoU reward handles precise-overlap cases.

Results: Scaling Works

Across seven benchmarks, TimeLens2-2B (a 2-billion-parameter variant) outperforms all same-sized baselines on every single benchmark. The larger 4B and 8B models reach state-of-the-art, beating open-source models up to 397B parameters—a striking efficiency gain.

Relative to their Qwen3-VL backbones (the base video-language models), the three variants improve by 14.2, 13.0, and 18.1 mIoU points, respectively. That's substantial: a 2B model with the right training strategy outperforms much larger competitors.

Why It Matters

Most prior work forces temporal grounding into single-interval or sequence-prediction frameworks. TimeLens2 shows that explicitly modeling the task as set prediction—through data curation and reward design—unlocks better generalization. The Wasserstein reward is practical and differentiable, avoiding the discrete matching headaches that plague earlier approaches. For real-world applications like video search and evidence retrieval, knowing all supporting moments (not just the first or longest) is essential.