Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

The Problem: RL Post-Training Wastes Annotations

Video multimodal large language models (MLLMs)—systems that understand video and language together—need reinforcement learning (RL) to improve performance on tasks like temporal detection and video segmentation. But current RL methods are sample-inefficient: they generate many rollouts (candidate model responses) per prompt, score them, and use only the scores to train. This requires expensive chain-of-thought reasoning and still produces few high-quality training signals per annotation. The authors identify a simple oversight: existing methods throw away the annotations themselves after scoring.

Core Innovation: Oracle Rollouts as Direct Training Targets

OraRL treats each human annotation as an oracle rollout—a guaranteed good response that directly supervises the policy. This sounds straightforward but creates a subtle problem called advantage inversion. In standard RL, the advantage measures how much better a rollout is than the baseline (average performance). If you add a high-reward oracle to the on-policy group, it raises the baseline so much that previously positive examples become negative, inverting their training signal and breaking optimization.

The authors solve this with a decoupled advantage estimator. Instead of using one baseline for all rollouts, they compute the baseline from policy-only rollouts (excluding the oracle), and separately modulate the oracle's contribution via an oracle-policy gap. This gap determines both a directional gain and a detached oracle advantage. The oracle always provides positive signal, while the policy rollouts maintain a meaningful comparison baseline. Additionally, sign-balanced pruning keeps only the oracle and one strong rollout of each sign (positive and negative advantage), drastically cutting the number of forward passes needed.

Why It Matters: Speed and Scalability

The efficiency gains are substantial. OraRL requires 2.2x the compute of supervised fine-tuning (SFT) versus 4.9x for GRPO, a competing method that uses chain-of-thought. More critically, OraRL works without chain-of-thought generation: Video-ORA-9B decodes in 130 ms instead of 4,780 ms—a 37x speedup—while achieving better results.

Performance across standard benchmarks:

• Temporal detection (temporal mIoU): 62.5 → 66.0
• Tracking (AO): 73.0 → 78.2
• Segmentation: 64.3 → 70.4
• Spatial-intelligence average: 51.0 → 56.1

On the VSI-Bench leaderboard, OraRL scores 73.1 compared to GPT-5 (55.0) and Gemini-3-Pro (55.1).

The approach scales smoothly from 0.8B to 9B parameters and handles up to 100k prompts, outperforming GRPO at larger scales. By treating annotations as actionable rollouts rather than passive scores, the authors unlock both sample efficiency and computational tractability—critical for making video MLLM post-training practical at scale.