On-Policy Self-Distillation without Any Supervision

The Core Claim

Self-distillation for large language models usually requires external ground truth, feedback systems, or bigger teacher models. This paper shows that's unnecessary: a model can improve itself using only its own outputs, by leveraging internal agreement patterns. The proposed method, U-OPSD (unsupervised on-policy self-distillation), matches or beats supervised approaches across math benchmarks without any external labels.

How It Works

The mechanism is elegant. U-OPSD samples multiple independent outputs from the model on the same problem. It then applies a majority-vote threshold: if enough samples agree on an answer, that becomes a "pseudo solution"—a candidate the model thinks is likely correct, even without ground truth verification.

The key insight is disagreement. The model now focuses on cases where it produced varied answers. It conditions its distribution on the pseudo solution (essentially, "given we think this is right") and then distills itself on the completions that diverged from majority opinion. This forces the model to understand why certain outputs were wrong relative to what the model itself deemed most probable. It's learning to correct confident mistakes—places where it generated multiple different answers but one emerged as most common.

This is genuinely unsupervised: no human labels, no external environment, no larger reference model. Only the model's own generation distribution and internal consistency patterns.

Why It Matters

The results are striking. On five mathematical reasoning benchmarks (AIME24, AIME25, HMMT25, MATH500, AMC23), U-OPSD improved base Qwen3 models by 8.5% at 4B scale and 10.7% at 8B scale. More impressively, it outperformed OPSD—the supervised method using ground-truth labels—by 3.2% and 2.3% on average at those scales. In thinking mode (where the model generates intermediate reasoning), U-OPSD stayed competitive with OPSD and beat GRPO (another supervised baseline) by 0.7–1.1%.

This matters for practical scalability. Obtaining ground-truth labels for every reasoning step is expensive and doesn't scale infinitely. Supervised feedback methods are narrow—they assume access to an oracle. U-OPSD flips the model's own uncertainty into a learning signal. Where the model is internally inconsistent is where it should improve.

The approach generalizes across benchmarks, model scales (4B to 8B), and training configurations, suggesting it captures something fundamental about how models can self-improve. It also sidesteps the cold-start problem: you don't need a teacher; the student teaches itself.

The unsupervised frame is philosophically important for AI safety and scaling. It means better models don't require proportionally better supervision. A model trained on its own consistency can refine reasoning without external oracles—an ingredient for autonomous improvement in open-ended domains where ground truth is ambiguous or unavailable.