- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 3
A conversation between
Parameter Exploration for RLVR via Variational Learning
§02
Snippets
-
Parameter-space exploration—sampling different policies from a posterior—enables complementary control over exploration beyond action-space methods like temperature scaling.
This reorders which tokens appear, not just their probabilities, unlocking exploration strategies that action-space methods cannot reach.
-
Perturbed Parameter Policy Optimization (3PO) samples multiple policies and groups rollouts for reward estimation, consistently reducing zero-advantage groups and malformed outputs versus baseline methods.
Fewer wasted rollouts and training instabilities mean more efficient learning from the same compute budget.
-
3PO improves average downstream performance on math reasoning and code generation at near-identical FLOPs cost to standard GRPO.
The efficiency gain suggests parameter-space exploration is a practical lever for RL-trained LLMs, not just a theoretical curiosity.
§03
Synthesis
The Core Finding
Exploration in large language model (LLM) reinforcement learning matters—but not just in the way most methods handle it. The authors show that sampling different policies (parameter sets) during training, rather than only adjusting output randomness within a single policy, leads to better and more stable learning. Their method, Perturbed Parameter Policy Optimization (3PO), consistently outperforms the standard GRPO baseline on math and code tasks while using nearly identical computational resources.
Why Action-Space Exploration Falls Short
Current LLM RL methods like GRPO typically control exploration through temperature scaling—essentially turning up the randomness in token selection. But this approach has a critical limitation: it cannot change the order in which tokens are ranked by the model. A token that the policy assigns low probability to stays low-probability, no matter how much you increase temperature. This constrains what solutions the model can discover during training.
The authors' insight is that exploration should happen at the parameter level instead. By sampling different copies of the policy—each with slightly different weights—each rollout can follow a genuinely different trajectory through the solution space. Some parameter samples might rank tokens differently, opening up entirely new reasoning paths.
How 3PO Works
The method is straightforward in concept: during training, generate multiple rollouts by sampling different policies from a learned posterior distribution. Rather than running one forward pass per training step, run several with different parameter perturbations. These diverse rollouts then feed into reward estimation and policy updates.
The family of 3PO methods varies along two dimensions: how you sample policies (e.g., different perturbation magnitudes or distributions) and how you group rollouts for advantage estimation (do you average rewards across all samples, or estimate advantages within groups?). The abstract doesn't detail the specific sampling strategies tested, but the framing makes clear that controlling policy diversity is the key lever.
The Results
On OLMo-3-1025-7B and Qwen2.5-Math-7B, 3PO variants beat GRPO on downstream math reasoning and code generation tasks. The improvements come at negligible additional computational cost—the authors note near-identical FLOPs. Beyond raw performance gains, 3PO produces fewer "zero-advantage groups" (batches where the model receives no learning signal because all rollouts score equally) and fewer malformed or incorrect outputs during training. This suggests the method not only explores better but trains more robustly.
Why It Matters
The work reframes exploration in LLM RL as a parameter-space problem, not just an action-space one. This is conceptually important—it suggests that diversity in how the model thinks, not just randomness in what it outputs, drives learning. For practitioners, it's a simple, low-cost lever: existing RL pipelines can incorporate policy sampling without major architectural changes. The consistency across two different 7B models and two task families hints that the finding generalizes beyond the specific experiments shown.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.