SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation

The Problem with Standard Teacher-Student Learning

When a large language model (teacher) guides a smaller model (student) on tasks like mathematical reasoning, the standard approach—reverse-KL divergence training—forces the student to match the teacher's exact probability distribution over next tokens. But this misses a critical insight: the teacher is often uncertain for good reason. Multiple different continuations might lead to correct answers downstream. The student doesn't need to copy the teacher's uncertainty; it needs to learn which options actually work.

Current methods can't distinguish between two very different uncertainty profiles: one where the teacher concentrates probability among three plausible continuations, and another where it spreads probability thinly across hundreds of low-probability tokens. Both look like "high entropy," but they're pedagogically different. The student benefits from focusing on the few promising paths in the first case, not all paths in the second.

Even worse, local signals—what the teacher prefers at a single step—don't guarantee the student will solve the problem end-to-end. A student might follow a teacher-approved token but then derail later. Standard distillation ignores this downstream performance data.

How SPOT Works

SPOT (Sparse Probing and Outcome-calibrated Targets) makes two intertwined decisions: where to query the teacher for detailed guidance, and what targets to use for training.

The method runs three phases in sequence:

Acquisition selects which positions (steps in a reasoning trajectory) deserve expensive teacher queries. A position-level score combines three signals: normalized teacher entropy (is the teacher uncertain?), the probability mass captured by the top-k candidates (is that uncertainty concentrated?), and student–teacher mismatch (does the student differ from the teacher here?). This budget-aware scoring prioritizes positions where the teacher is uncertain and concentrated and where the student disagrees.

Exploration verifies which of the teacher's top-k suggestions actually work. Instead of trusting the teacher's local probability alone, SPOT uses a verifier (a classifier trained to judge solution quality) to score student-generated continuations that follow each teacher-suggested token. This reveals which candidates lead to downstream success.

Exploitation builds training targets from verified outcomes. Rather than matching the teacher's original distribution, SPOT constructs a closed-form target that upweights candidates with better verifier scores while staying anchored to the teacher distribution via KL regularization. This balances learning from successful paths (high exploitation) without drifting too far from the teacher's initialization (regularization).

The key insight is coupling: the acquisition phase identifies which positions matter most (concentrating computational budget), and the exploration phase ensures targets actually predict success rather than just mimicking local teacher preferences.

Why It Matters

Experiments across multiple reasoning benchmarks show SPOT improves student performance while using less teacher supervision than baseline on-policy distillation. By probing selectively and calibrating targets to downstream outcomes, the method sidesteps the false choice between matching the teacher exactly and ignoring teacher guidance entirely. This matters for scaling language models cost-effectively: better distillation means weaker students can solve harder problems with less compute and fewer calls to expensive teachers.