Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

The Core Claim

Rather than retraining or fine-tuning language models to improve agent performance, the authors show that you can continuously rewrite the harness—the executable code scaffold that wraps the model—to achieve significant gains on specific tasks. Using a frozen LLM backbone, their framework called Hierarchical Self-Improvement (HSI) evolves the task harness through automated feedback, yielding improvements of 15–39 percentage points on moderate-difficulty benchmarks while maintaining strong generalization to unseen task variants.

How It Works

The system operates at three nested levels. At the bottom, a task harness executes the actual task—it contains the logic for tool-calling, state transitions, and prompt formatting for a specific problem family. Above it sits an evolver that rewrites this harness code based on environment feedback (success/failure signals). At the top, a meta-evolver refines the evolver's own strategy code, all while the core LLM remains frozen.

The thinking-on/off design is key to isolating what matters: the frozen model runs in "thinking-off" mode during task execution (faster, simpler), then switches to "thinking-on" mode during self-modification phases when it redesigns the harness. This design choice makes it clear that improvements come from harness evolution, not from reasoning-time scaling.

The process is concrete: after each task attempt, the environment provides a reward signal. The evolver uses this signal to propose modifications to the harness—reordering steps, adding intermediate checks, restructuring tool calls, changing state representations. The meta-evolver then learns which modification strategies work best and improves the evolver's code accordingly. Critically, a task-injection seam keeps harnesses modular and swappable across iterations.

What Works and Why It Matters

On BALROG benchmarks using DeepSeek-V4-Flash-Preview as the backbone, HSI delivers consistent gains: +39.3% on BabyAI, +33.0% on Crafter, +25.0% on TextWorld, and +15.0% on MiniHack. The framework also generalizes well to held-out task variants (0.98 and 1.00 accuracy on unseen splits of BreakStop and GoTo respectively).

However, the authors identify clear limits. Harness evolution cannot overcome the frozen model's inherent capability gaps—on NetHack Legends Environment (NLE), where the backbone simply cannot solve problems, rewriting the harness yields no improvement. Similarly, improvements depend on feedback quality; weak or uninformative reward signals prevent effective evolution.

This matters because it offers a practical middle ground between manual prompt engineering (labor-intensive, brittle) and full model retraining (expensive, often infeasible). For deployed systems using frozen models—whether for cost, safety, or latency reasons—task-specific harness evolution becomes a viable lever for continuous improvement. The work also challenges the assumption that harnesses are static infrastructure; treating them as evolvable artifacts opens a new dimension for agent optimization that doesn't require touching the model itself.