Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

The Problem: RL Doesn't Flow Through Images

Current multi-modal models that mix text and image generation hit a wall with reinforcement learning. When a model reasons by writing text, then generating an image, then writing more text, existing RL methods only train the text parts. Images get stuck in supervised learning mode, acting like frozen surrogates. This breaks the chain—policy gradients can't travel backward through the full reasoning sequence, so the model never learns to generate better images for the sake of the downstream reasoning task. The authors show this leaves massive optimization potential on the table.

The Core Insight: One MDP for All Modalities

BRAID (Bridging inteRleAved multI-modal reasoning as a unified Decision process) reframes the problem: treat the entire interleaved reasoning chain—text, image, text again—as a single Markov decision process, the standard framework for sequential decision-making in RL. This sounds simple, but it's the breakthrough. Instead of optimizing text and images separately, a unified MDP lets both modalities share a single trajectory-level advantage score (a measure of how good the overall chain is) and each gets updated through its own native mechanism.

For text, that's the standard token-level policy gradient. For images, it's a diffusion-based gradient that nudges the denoising path toward better outputs. Crucially, both receive the same high-level signal about whether the full reasoning sequence worked.

Sharpening Credit Assignment with a Judge

Long multi-step chains struggle with credit assignment: if a final answer is wrong, which step caused it? An image that looked reasonable might have derailed the reasoning. BRAID brings in a vision-language model (VLM)—a model trained to understand both images and text—to score each intermediate image on how useful it is for the reasoning task. These intermediate rewards act as breadcrumbs, giving the model denser feedback at critical visual branches rather than waiting until the end. This addresses the classic deep RL problem of sparse rewards over many steps.

Why It Matters

The paper demonstrates gains on spatial reasoning tasks (where multi-step visual-spatial logic is required) and visual perception benchmarks. The consistency of improvements confirms that a unified MDP plus vision-grounded feedback is genuinely necessary; simpler patches don't cut it. This matters because interleaved reasoning—where models think and draw and think again—is becoming central to multimodal AI. Unlocking proper RL optimization for these systems is a step toward models that can plan and reason across modalities together, not in silos.