SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning

Self-Verification as a Learning Signal

SVR-R1 reframes a vision-language model's own doubt as training fuel. When the model answers a visual reasoning question, it simultaneously issues a binary verdict on its confidence. If it says "No, I'm unsure," the system gives it another turn to reconsider. This self-doubt becomes the mechanism that triggers refinement—and crucially, this refinement loop itself teaches the model to reason better, without needing external judges or annotators.

The framework operates within a reinforcement learning loop using GRPO (Group Relative Policy Optimization). For each query, the model generates an answer and a self-verdict using identical weights. A "No" spawns a second attempt; a "Yes" or hitting a turn limit freezes the output. The final answer then receives an outcome-based reward (correct or incorrect), which backpropagates through the entire multi-turn trajectory. Critically, the model learns from both successful reasoning and failed self-corrections—the signal is just the ground truth answer, not external verification scores.

Why This Matters

The key insight emerges in the training dynamics: as the framework iterates, models rely on fewer verification turns but achieve higher test accuracy. This signals that the gap between "I need to verify this" and "I'm confident enough to answer" shrinks as the policy internalizes self-correction. The model isn't just learning to second-guess itself; it's learning when not to. By episode's end, it picks the most confident path without redundant reconsideration.

Evaluations on vision-language reasoning benchmarks show substantial accuracy gains over standard GRPO baselines, though the abstract doesn't specify exact metrics or datasets. The asynchronous multi-turn rollout framework ensures computational efficiency—the model doesn't need auxiliary critics, reward models, or human feedback, only its own verdicts and eventual outcome labels.

This bridges two usually separate areas: inference-time self-refinement (popular in language models, less explored for vision-language tasks) and RL training for multimodal systems. Most prior work either adds external supervision or treats refinement as a fixed post-hoc step. SVR-R1 bakes refinement into the learning objective itself, letting the reward signal directly optimize when and how to reconsider.

The result is a simpler pipeline than alternatives requiring critic networks or supervised verification labels, yet empirically stronger than models that reason once and commit. The authors plan to release the code, signaling confidence in reproducibility and practical utility for future vision-language model research.