CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation

The Core Problem: Student Models Collapse When Learning from Teachers

Standard knowledge distillation—where a small "student" model learns by imitating a large "teacher" model—works well for simple tasks but breaks down for reasoning. The authors identify three specific failure modes: fresh students initially assign almost zero probability to the teacher's preferred tokens (cold-start collapse); the training schedule for blending two loss types ignores whether the student is actually learning to cover the right outputs (state-agnostic divergence); and binary reward signals (right/wrong) throw away useful information from answers that are nearly correct.

CADENCE fixes all three problems with targeted mechanisms, achieving 69.8% accuracy on GSM8K math problems with a 0.5B student distilled from a 1.5B teacher—a gain of 21.1 percentage points over the untrained student and closing 63% of the gap to the teacher's performance.

How CADENCE Works

The framework starts with DRIFT, a per-token mechanism that balances forward-KL (teacher-favoring) and reverse-KL (student-favoring) losses. Unlike prior work that adjusts this blend globally over time, DRIFT's six extensions make the blend adaptive to what the student has actually learned:

• COVA schedules the forward-to-reverse transition based on coverage—how well the student matches the teacher's token probability distribution—rather than on a fixed schedule.
• FTB concentrates learning effort where it matters most: high-entropy positions in the teacher's outputs, using a globally-normalized entropy reference to avoid token-by-token bias.
• CCD replaces the binary pass/fail signal with a dense reward that gives partial credit when the model's answer is numerically close but wrong, capturing gradations of nearly-correct reasoning.
• LAP reinforces correct rollouts while preferring shorter ones, avoiding verbose solutions.
• EMR calibrates the student's entropy (confidence distribution spread) to match the teacher's via a regularizer, preventing overconfidence.
• BSD adds a bootstrapped self-distillation phase where the student distills from its own improved version.

All components operate on trajectories (sequences of tokens) sampled from the student during training, not from a fixed dataset.

Why This Matters

The results are striking: on MATH-500, a harder benchmark, the 0.5B student reaches 72.1% accuracy with a 3B teacher, closing 76% of the performance gap. More importantly, all experiments run on a single consumer machine (Apple Mac Studio), not a datacenter. This democratizes reasoning model distillation—practitioners can now train compact, capable models without enterprise-scale hardware.

The unified framework also clarifies that reasoning distillation requires simultaneous fixes to initialization, scheduling, and reward structure. None of these pieces alone solves the problem; each compounds the others. By addressing all three systematically, CADENCE shows that on-policy distillation can be principled and practical.