EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

## The Problem: RL Rollouts Hit a Speed Wall

Reinforcement learning on large language models requires generating many sample responses (rollouts) during training. The bottleneck is clear: autoregressive decoding—where tokens come out one at a time—is slow, and a few unusually long generations can stall the entire batch. Speculative decoding (SD) is a proven speedup technique for serving fixed models: a fast drafter generates token sequences, and a verifier checks them in parallel, accepting or rejecting in bulk. But applying it directly to RL training fails because two problems compound: the target policy keeps changing as it trains, so any fixed drafter becomes increasingly misaligned with what the model now outputs; and as batches shrink during decoding, the system shifts from compute-bound to memory-bound, wasting the parallel compute that SD relies on.

## How EfficientRollout Works

The authors' core insight is that RL rollouts need a *coupled* drafter and *system-aware scheduling*, not a generic speedup.

**Self-speculative drafting**: Instead of training a separate drafter, EfficientRollout quantizes the target model itself—compressing weights to lower precision—to create a lightweight draft model that stays synchronized with the evolving policy. This eliminates the misalignment problem: as the policy trains, the drafter automatically adapts because it's derived from the same weights.

**System-aware toggling**: EfficientRollout doesn't always use speculation. It learns when to turn SD on or off based on real system metrics—whether the batch is compute-bound or memory-bound. When batches are small or memory-bound, parallel verification can't exploit idle compute anyway, so turning off SD saves orchestration overhead. The framework pairs this toggle with draft-length adaptation, which adjusts how many tokens the drafter predicts based on its current acceptance rate (how many drafts the verifier actually accepts). High acceptance means the drafter is tracking the policy well, so longer drafts are safe.

## Results and Impact

On RL rollout generation, EfficientRollout achieves up to **19.6% latency reduction** compared to accelerated autoregressive baselines, and up to **12.7% end-to-end training latency reduction**. Critically, final model quality is preserved—the speedup doesn't degrade RL training outcomes.

The work matters because RL-based post-training (chain-of-thought reasoning, agentic behavior) has become central to LLM scaling, and rollout latency is now a primary training bottleneck. Current serving accelerators don't handle the unique constraints of RL: evolving policies and shrinking batches during generation. By coupling the drafter to the policy and making speculation conditional on system state, EfficientRollout makes speculative decoding practical for the RL setting without requiring expensive drafter pretraining or online fine-tuning. This is a systems contribution that directly addresses a real training bottleneck, not an isolated serving optimization.