Latent On-Policy Self-Distillation

The Core Innovation

Existing self-distillation methods for AI agents rely on hand-crafted privileged information—predefined answers, skills, or feedback structures—that a teacher uses to guide a student policy. This bottleneck limits scalability and forces humans to design what the teacher should know. The authors propose Latent On-Policy Self-Distillation (LOPD), which instead learns what privileged context the teacher should use directly from the agent's own experience, eliminating the need for designer-specified artifacts.

How It Works

LOPD operates in two coupled loops. The student generates trajectories (sequences of actions and observations) from task and interaction history. Meanwhile, the teacher learns to construct its own privileged context by retrieving relevant past experiences and composing them into continuous latent tokens—learned embeddings that capture patterns the agent discovers. These latent tokens condition the teacher to provide dense, token-level supervision at every step of the student's trajectory.

The key technical contribution is the privileged-margin objective, a regularization mechanism that stabilizes learning of these latent tokens, preventing the teacher from becoming either uninformative or exploitably easy.

The process is end-to-end learnable: as the agent accumulates experience, both the student policy and the latent token generator improve together, without requiring humans to specify what form privileged information should take.

Why This Matters

Empirical strength: Across two domains—agentic tool use (like calling APIs) and code generation—LOPD outperforms established on-policy self-distillation baselines (OPSD, SDPO, Skill-SD) and RLVR, a recent retrieval-augmented RL method.

Sample efficiency: LOPD achieves better results than GRPO and Skill-SD using less than 30% of their rollout budget. In reinforcement learning, sample efficiency directly translates to reduced computational cost and faster training.

Fundamental shift: Most prior work assumes the teacher's knowledge structure is fixed and provided upfront. LOPD inverts this—it asks what the teacher should know, then learns it. Ablations confirm this learnability is necessary for the performance gains.

This addresses a real scalability problem: as agents tackle harder problems, hand-crafting privileged information becomes impractical. A system that discovers its own useful internal representations is more likely to scale to open-ended self-improvement.

The paper positions itself as a step toward continual, self-directed agent evolution—agents that internalize lessons from their own experience without external scaffolding.