Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

The Core Problem and Solution

Current reinforcement learning approaches for improving reasoning in language and vision models rely heavily on ground-truth labels—expensive annotations that become nearly impossible to obtain as models surpass human evaluation capability. Self-rewarding RL sidesteps this by having models generate their own feedback, but it has a fatal flaw: models reinforce their own mistakes, creating feedback loops that collapse into homogenized, low-quality responses.

The authors demonstrate that unsupervised reasoning can emerge when multiple independent models critique each other instead. This insight—that diversity breaks destructive feedback cycles—forms the basis of Co-RL.

How Co-RL Works

Co-RL trains a cohort of separate, parameter-sharing models simultaneously. Rather than a single model rewarding itself, each model in the group receives reward signals from its peers. Crucially, the cohort is intentionally heterogeneous: it includes models of different sizes, different architectures, and even receives rephrased versions of the same training samples to increase perspective diversity.

The mechanism is straightforward: when one model generates a response, other models in the cohort evaluate it. This peer feedback becomes the reward signal for RL optimization. Because the models have different inherent biases—due to size differences, architecture differences, and varied input phrasings—they disagree on which responses are good. These disagreements prevent any single model's mistakes from being unanimously reinforced. The diversity acts as a natural error-correction mechanism.

Why This Matters

The results are substantial. Across seven text-only benchmarks, Co-RL achieves average gains of 3.0–8.6% over base language models without using any labeled data. On four multimodal vision-language benchmarks, improvements reach 2.3–7.2%. Critically, these gains match or exceed supervised baselines that do use ground-truth labels.

Beyond raw performance, Co-RL preserves behavioral diversity—responses remain varied rather than collapsing into repetitive patterns—and maintains stable training without the catastrophic failures seen in self-rewarding approaches.

The practical implication is significant: as reasoning tasks become too complex for humans to verify (e.g., novel mathematical proofs, complex strategic decisions), this framework provides a scalable alternative to expensive annotation. By leveraging cohort disagreement rather than consensus, Co-RL transforms a liability of multi-agent training—that models might disagree—into an asset.

The work spans both text-only and multimodal settings, suggesting the insight generalizes across modalities. The authors release code, enabling reproducibility and extension.