Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

The Core Innovation

Most LLM post-training methods force the model itself to explore and learn simultaneously—a costly, inflexible approach that locks optimization signals to a single model and moment in time. This paper proposes PUST (Proxy-guided Update Signal Transfer), which decouples these two tasks: use a cheap, lightweight proxy model to explore and discover good behaviors, then transfer those learnings to the actual model. The result is a modular, reusable post-training pipeline that cuts computational cost and allows signals to be cached and shared across models.

How It Works

The method operates in three stages:

Proxy exploration: Rather than optimizing the primary (expensive) model directly, PUST trains a small proxy model on the same task. This proxy explores the reward landscape cheaply—think of it as a scout finding high-value behaviors at a fraction of the cost.

Update-signal extraction: The key insight is extracting relative improvement signals—the directional shift from the proxy's initial to optimized state—rather than copying the proxy's final policy distribution. This relative signal captures "what changed" rather than "where we ended up," making it transfer-friendly.

Signal transfer: The primary model receives this directional guidance and aligns its own policy accordingly. Crucially, since the signal is relational rather than absolute, a weak proxy can still guide a stronger model upward, and one proxy's learnings can benefit multiple primary models without modification.

The authors evaluate PUST on Qwen3-family models across math and code generation tasks. They show that update signals from substantially weaker proxies robustly improve stronger primary models. The framework achieves asynchronous generation and caching of signals—optimization happens offline and signals are reused, rather than requiring live exploration on the primary model.

Why This Matters

Current post-training is monolithic and expensive: the model being trained must simultaneously explore new behaviors and learn from them in real time. This creates three bottlenecks. First, computational cost—exploration on large models is wasteful. Second, rigidity—signals are tied to a specific model and cannot be reused or transferred. Third, inflexibility—weak-to-strong improvement (using a small model's insights to guide a larger one) is difficult because absolute policy distributions don't transfer cleanly.

PUST solves these by inverting the dependency: make the proxy do the exploration work, extract only the relative improvements, and transfer those. This is cheaper (proxy is small), modular (signals cached and reused), and naturally supports cross-model transfer (relative signals don't require the receiver to match the sender's scale or architecture).

The practical upshot is that post-training becomes a supply-chain process: proxies generate reusable optimization signals offline, primary models consume them asynchronously. Organizations can build signal libraries, apply them to multiple models, and avoid redundant expensive exploration. For researchers, the framework suggests that policy transfer doesn't require matching distributions exactly—directional information alone can guide improvement, a finding with implications beyond post-training.