Best Practice Critic Optimization

The Problem: Why Critics Are Tricky in Language Model Training

Group-based reinforcement learning methods like GRPO sidestep a fundamental challenge: they avoid training a value function (or "critic") that estimates how good a decision is. Instead, they sample multiple responses per prompt and compare them directly to compute advantages—a relative ranking that guides policy updates. This works, but it's sample-inefficient. A well-trained critic could estimate advantages from just one response, slashing sampling costs. The catch: standard critic training in this setting is notoriously unstable, causing policy learning to fail or plateau.

The authors identify why and propose Best Practice Critic Optimization (BPCO), a deliberate recipe that stabilizes critic-based training without sacrificing performance.

What Makes BPCO Work

BPCO combines five design choices, each validated through ablation:

1. DPPO (Distributed Proximal Policy Optimization) — A variant of PPO that constrains policy updates to stay close to the previous policy, reducing destructive swings.

2. Bounded value predictions — The critic's output is constrained to the actual range of rewards observed, preventing it from predicting impossible values that destabilize learning.

3. Monte Carlo value targets — Instead of bootstrapping (using the critic's own next-state prediction), the authors use actual returns from trajectories. This removes circularity and noise from the critic's self-estimates.

4. Unnormalized policy advantages — The advantage signal fed to the policy is not normalized (standardized to zero mean and unit variance), which preserves the scale of information the critic computes.

5. Length-adaptive generalized advantage estimation (GAE) — A technique that blends one-step and multi-step advantage estimates, adjusted for token sequence length to handle variable-length responses fairly.

Critically, the critic sees information the policy doesn't—like a reference answer or grading rubric—which the policy must infer from the reward signal alone. This asymmetry boosts critic capacity without leaking privileged information.

Results and Why It Matters

On mathematical reasoning tasks (across 1.5B to 30B-A3B parameter models), BPCO consistently improves over a strong critic baseline and matches or exceeds GRPO's performance while sampling only one response per prompt. This is significant: it delivers group-relative advantage's stability and quality using a fraction of the samples.

The same recipe works with rubric-based rewards, suggesting generality beyond the specific test setting.

The core insight is pragmatic: instability in critic-based RL isn't inevitable. It reflects misaligned design choices—unbounded predictions, self-bootstrapped targets, normalized advantages—that amplify noise and divergence. By addressing each, the authors show a critic becomes a reliable, sample-efficient alternative to group sampling.

For practitioners, BPCO offers a concrete blueprint: it's not a novel algorithmic breakthrough, but a careful engineering result that makes critic-based training competitive on large language models. Code release lowers the barrier to adoption.