Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

One Layer Is Often Enough for RL Fine-Tuning

Reinforcement learning (RL) post-training of large language models typically updates all transformer layers uniformly, operating on the implicit assumption that every layer contributes equally to performance gains. This paper demonstrates that assumption is wrong: a single transformer layer can recover most or all of the improvement from full-parameter RL training.

The authors conduct a systematic layer-wise study across seven models (Qwen3 and Qwen2.5 families), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and diverse tasks including mathematical reasoning, code generation, and agent decision-making. Their key finding is that RL gains concentrate heavily in a small subset of layers—often just one—rather than distributing evenly across the network.

How It Works and What Pattern Emerges

The researchers measure "layer contribution": the fraction of full-parameter RL improvement recovered when training a single layer in isolation while keeping all others frozen. They then rank layers by contribution and examine where in the transformer stack the high-contribution layers cluster.

The pattern that emerges is remarkably consistent across models, tasks, and algorithms: high-contribution layers concentrate in the middle layers of the transformer stack, while layers near the input (early) and output (final) ends contribute substantially less. This structural ranking remains strongly correlated even when shifting between datasets, task types, model families, or the specific RL algorithm used.

Crucially, the authors find that training only the most important layer(s) can match or occasionally exceed the performance of updating the entire model. This suggests that full-parameter RL training may be inefficient—it updates many parameters that don't meaningfully influence performance gains.

Why This Matters

The practical implications are significant. If a single layer captures most RL gains, researchers and practitioners can:

• Reduce computational cost during RL fine-tuning by updating only a handful of middle-layer parameters instead of all weights
• Simplify optimization by reducing the number of parameters and search space during adaptation
• Improve training stability by focusing gradient updates on layers that matter most

More fundamentally, this work reveals that RL adaptation is not uniformly distributed across the model. The concentration of gains in middle layers suggests those layers play a special role in behavioral change during RL—possibly bridging high-level semantic understanding (early layers) and task-specific action generation (late layers). Understanding why middle layers dominate could inform better architecture design or more efficient fine-tuning strategies.

The consistency of the pattern across multiple model families, algorithms, and tasks hints that this is a fundamental property of transformer-based RL, not an artifact of specific implementation choices. This opens questions about whether similar structure appears in other fine-tuning paradigms or whether it generalizes to other model architectures.