LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

The Core Problem

Reinforcement learning (RL) has become the go-to approach for training coding agents—systems that write code to solve software engineering tasks. These agents rely on long-running "harnesses" (execution environments) that manage tools, repository contexts, and feedback loops. The catch: these harnesses were never designed for policy-gradient training, the standard RL method for optimizing language models. The result is a fundamental mismatch. Harnesses crash unpredictably, corrupt reward signals through incentive exploitation (reward hacking), and drift in behavior between training and deployment because the policy being optimized doesn't see what actually runs.

How LEGO-RL Works

The authors propose three tightly integrated solutions that wrap around native harnesses without modifying them internally.

Faithful optimization tackles the signal corruption problem. The framework intercepts the raw token stream from the language model before the harness processes it, capturing generation data at the token level. Crucially, it recomputes log-probabilities (the likelihood scores needed for policy-gradient updates) on the trainer side, not the harness side. This guards against data loss when harnesses compress or re-serialize outputs. The in-process LLM proxy acts as a bridge, ensuring the policy updates reflect what was actually generated, not a lossy reconstruction.

Reliable execution mitigates reward hacking and crashes through sandbox orchestration. The framework uses image caching (pre-baked execution environments) and stage-wise defenses—layered safeguards that catch invalid states before they corrupt the reward signal. Scalable sandbox management means training can run many parallel rollouts without cascading failures.

Observable training adds transparency. An integrated plugin automatically validates and monitors runs, feeding data into a Live UI that lets researchers drill down into individual trajectories. This closes the visibility gap that typically plagues large-scale RL experiments.

Why It Matters

The authors trained a sparse mixture-of-experts model (Qwen3.5-35B-A3B) using a policy-gradient algorithm called GSPO across three different real coding-agent harnesses. Results are substantial: on SWE-bench Verified (a benchmark for software engineering tasks), performance jumped from 64.0% to 70.4% on OpenHands SDK, 62.4% to 68.2% on Claude Code, and 57.2% to 66.6% on OpenCode. Critically, they maintained a rollout-training probability correlation above 0.99, meaning the behavior during rollouts stayed aligned with what the policy was optimizing—solving the train-inference discrepancy.

The insight is practical: rather than redesign harnesses for RL (costly and fragile), LEGO-RL acts as a translation layer. It extracts clean training signals from messy real-world execution environments and orchestrates them safely at scale. For teams running coding agents in production, this removes a major bottleneck: you can now train directly against your existing tools and systems without choosing between fidelity and stability.