The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

The Core Problem: Training and Inference Live in Different Worlds

Large language models in reinforcement learning training use separate engines for generation (inference) and learning (training) to balance speed and precision. The catch: even with identical model parameters, these two engines produce different probability distributions for the same text sequences. This creates a persistent mismatch that poisons the training signal—the model optimizes one policy during training but deploys a different one in practice.

Prior work tried to fix this by reducing off-policyness (the technical term for when your training data doesn't match your deployment setting). But the authors identify a subtler, overlooked problem: improving the training policy doesn't guarantee improving the inference policy. A parameter update that looks good in the training engine might actually hurt the policy you actually use when generating text. The paper argues this objective misalignment is the real culprit behind training instability and collapse in LLM RL.

The Solution: Monotonic Inference Policy Improvement (MIPI)

The authors propose MIPI, a principle stating that updates should improve the inference policy directly, not just the training policy. They instantiate this with MIPU (Monotonic Inference Policy Update), a two-step framework:

Step 1: Generate candidates. Create candidate parameter updates using the training engine, sampled in a way that references the sampling distribution (hence "sampler-referenced").

Step 2: Selective acceptance. Before applying updates, check whether they actually improve the inference policy. The authors use an "inference-side gap proxy"—a measurable signal that approximates how well the update will perform at inference time—to decide which candidates to accept. Only updates predicted to help inference policy get synchronized to the model.

The intuition is straightforward: instead of blindly trusting that training improvements carry over, explicitly validate that inference performance improves before committing to changes. This creates a feedback loop where the training process is forced to align its objectives with what matters at deployment.

Why This Matters

Training instability is a well-known pain point in LLM RL. Models collapse, performance plateaus, or oscillates wildly. Current fixes patch the off-policyness symptom without addressing the goal misalignment root cause. MIPU directly targets the real issue: ensuring every training step benefits the deployed model.

The experiments tested on two model scales under "high mismatch" conditions (where training and inference engines diverge significantly). Results show MIPU improved both average reasoning performance and training stability compared to baselines. The practical upshot: more reliable, predictable RL training for LLMs with fewer collapsed runs and faster convergence.

This work reframes what "good training" means in the presence of infrastructure constraints. Rather than fighting the mismatch, MIPU accepts it exists and builds safeguards to prevent training-inference divergence from breaking the system. For practitioners scaling LLM RL, this offers a concrete alternative to ad-hoc stability tricks.