Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

The Core Insight

Standard reinforcement learning for reasoning models—like GRPO—only grades whether the final answer is correct. This creates a perverse incentive: on hard problems, models learn to write longer solutions rather than better ones, since the reasoning process itself is never evaluated. Agon flips this by making two models compete directly: one drafts a solution while the other reads it and solves the problem independently. Each model is rewarded for out-reasoning its rival, so good reasoning gets implicitly graded during training—no process labels or reward model needed.

How It Works

The training loop alternates roles between two comparable but behaviorally different models. In each round, Model A drafts a solution to a problem, and Model B reads that draft while solving the problem itself. The model that produces the correct answer wins the round. Because Model B has access to Model A's reasoning before answering, Model B has an unfair advantage unless Model A reasoned so well that the draft genuinely helped. To overcome this handicap and actually win, Model A must think deeply, not just write more words.

Both models face continuous pressure: each is optimized via RL, so each becomes progressively stronger, and each in turn faces a stronger rival. Unlike single-model RL, which trains against a static baseline, both participants improve together, creating an escalating challenge that pushes reasoning harder.

At inference, the trained pair deploys as a two-stage cascade: one model drafts, the other reads the draft and produces the final answer. No ensemble voting or extra compute beyond two forward passes.

Why It Matters

The empirical gains are substantial. On the hard split of DeepMath using Qwen3, Agon roughly doubled GRPO's pass@1 score—approximately eight times larger than the improvement from an untrained Mixture-of-Agents baseline applied to the same base model. The pattern held on competitive-programming code and generalized across model families (Qwen3.5, Gemma 4), suggesting the mechanism is robust and not model-specific.

The conceptual contribution is equally important. Current reasoning RL methods have a structural blind spot: they optimize for final answers only, leaving the actual reasoning process ungraded. This creates a gradient toward verbosity and padding rather than depth. Agon grounds the evaluation of reasoning in competitive performance against a rival that has transparency into your work. This is closer to how humans improve—by explaining ideas to skeptics or teaching peers who already know your argument. No ground-truth process labels or learned reward model are needed; the competition itself is the grading signal.

The authors note that both models currently communicate in text. The next frontier is reasoning together in latent space, which would eliminate the communication overhead and potentially unlock even tighter co-optimization. For now, Agon represents a practical path to reasoning models that think harder, not just longer—and a demonstration that competition between models can be more effective than solo RL for training reasoning.