Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

The Problem with Entropy-Based Learning

Standard reinforcement learning methods that use verifiable rewards (RLVR)—where you can check if a generated response is correct—typically rely on entropy to shape learning signals. Entropy measures uncertainty: high entropy means the model is spread across many possible outputs, low entropy means it's confident. The assumption is simple: more uncertainty should be penalized because it signals confusion. But this breaks down in practice. A model can be uncertain usefully (exploring multiple valid solutions) or uselessly (genuinely confused about incorrect tokens). Entropy can't tell the difference, so it throws away learning signal that could distinguish right from wrong.

The authors' key insight is that token-level disagreement between two different generation processes reveals actual correctness without this ambiguity.

How Contrastive Policy Optimization Works

CPO uses two distributions side-by-side during training. The first is standard generation—what the model would normally produce. The second is reference-guided generation, where the model conditions on the correct answer while generating. The disagreement between these two distributions, measured using contrastive methods, pinpoints tokens where the model's default behavior diverges from what it would generate if it "knew" the answer.

This disagreement serves as a correctness signal: tokens where the model naturally agrees with the reference-guided version likely indicate correct predictions, while large disagreements flag problem areas. Crucially, this works at the token level, not just at the sequence level, enabling fine-grained learning.

A secondary insight: on-policy distillation (training against an external teacher model) turns out to be a special case of CPO where the reference-guided distribution is replaced by the teacher. This unifies two seemingly different approaches under one framework.

CPO also sidesteps the "zero-advantage problem"—a pathological case where entropy-based methods assign the same advantage to all tokens in a correct sequence, providing no gradient signal. Contrastive disagreement naturally assigns different advantages based on actual alignment with the reference.

Why It Matters

Experiments on multiple benchmarks show CPO substantially outperforms entropy-based RLVR methods. More interesting is the finding that correct and incorrect responses naturally partition into exploration and exploitation behaviors: incorrect responses push the model to explore alternatives (useful early), while correct ones encourage exploitation (useful later). Balancing both yields the strongest results.

The method generalizes well to out-of-domain benchmarks, suggesting the contrastive disagreement signal is robust. By moving beyond entropy as a proxy for correctness and directly measuring alignment between the model's natural behavior and its reference-guided behavior, CPO provides a more honest measure of what the model actually knows—enabling more effective reinforcement learning in settings where correctness can be verified.