Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

The Problem: Fixed Reward Weights Miss the Real Optimization Frontier

Multi-objective reinforcement learning for language models typically combines multiple reward signals—say, reasoning correctness and response length—into a single scalar by taking a weighted sum, then standardizing advantages across a batch. The authors expose a critical flaw: two rollouts with completely different reward profiles can end up with identical advantage values, and more crucially, the optimizer keeps pushing all objectives equally hard regardless of whether they're already solved.

Imagine training a math reasoner on two goals: getting answers right and keeping responses concise. Once conciseness is nailed (99% solved), you don't need more gradient budget there. Yet traditional methods still allocate equal effort to both. This wastes capacity that could improve the harder objective—correctness.

How SA-MRPO Fixes This: Adaptive Objective Reweighting

The key insight is saturation awareness. Instead of fixing reward weights upfront, the authors' method (SA-MRPO) does two things:

First, standardize independently. Each reward objective gets its own normalization rather than being mixed before standardization. This ensures rollouts with different reward profiles actually receive different advantage signals.

Second, estimate and discount by saturation. At batch level, the method estimates how "solved" each objective is—essentially: how close is the batch median to the best-case performance? Objectives near their ceiling get discounted; struggling objectives get full weight. Interestingly, saturation-aware reweighting can even flip the sign of a gradient update, not just shrink it, when an objective has saturated but would otherwise push in a harmful direction.

The authors call this batch-level saturation estimate the discount factor. High saturation (objective mostly solved) → low discount. Low saturation (lots of room for improvement) → full weight.

Why It Matters: Empirical Gains on Hard Problems

Across benchmarks:

• Math reasoning (AIME24, etc.): SA-MRPO beats the baseline (GDPO) on the harder correctness objective in 12 of 15 comparisons, with up to 5% improvement on AIME24.
• Adaptive reasoning (AMC23, etc.): 3.8% average gain, up to 9.2% on AMC23; all five benchmarks improve.
• Coding: Up to 2.3% pass-rate gain.

Critically, easier objectives stay near their already-satisfied levels—you don't sacrifice what's working. The reallocation is surgical.

This matters because post-training language models are increasingly multi-objective: they need to be correct and concise and safe. Fixed weight schemes are theoretically blind to saturation; they're optimizing a frozen problem when the actual problem shape changes as training progresses. SA-MRPO makes the optimizer responsive to that changing landscape, concentrating compute where it's most useful.