Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

The Core Finding

Rewriting the instructions you give to an LLM (called "harness optimization") can dramatically improve performance without changing the model itself. But doing this well requires evaluating the harness on many validation tasks repeatedly—a process that becomes wasteful as optimization progresses. Task-CoEvolve cuts evaluation costs by 80% while achieving the same final performance as exhaustive search by intelligently choosing which validation tasks to run at each iteration.

Why This Matters

LLM harness optimization is appealing because it's cheap compared to retraining. But the bottleneck is validation: you need to test each candidate harness on a full validation set to rank them and pick the best one. As the harness improves, many validation tasks become easy or stay hard consistently. Evaluating all of them every iteration wastes computation on tasks that no longer help you distinguish good harnesses from mediocre ones.

The authors' key insight is that tasks where different candidate harnesses disagree—where some solve it and others fail—carry the most information. Tasks that all harnesses solve or all fail don't tell you which candidate is actually better. By focusing evaluation on these "frontier" tasks, you maintain discriminative power while cutting total evaluation cost.

How Task-CoEvolve Works

The method has two main pieces:

Adaptive task selection. Rather than evaluating all tasks at each iteration, Task-CoEvolve samples tasks based on past outcomes. It uses variance-weighted sampling: tasks showing high disagreement among candidate harnesses get higher probability of being evaluated next. This directs compute toward tasks that actually matter for ranking harnesses. The sampling distribution adapts as the harness evolves and tasks shift from hard to easy (or vice versa).

Full-set score estimation. Sampling different task subsets across iterations could bias your comparisons—a harness might rank high simply because you tested it on easier tasks. Task-CoEvolve solves this by accounting for sampling probabilities when estimating the full-set performance from partial evaluations. This ensures fair ranking of candidates even though different iterations evaluate different subsets.

Together, these let the method run fewer total evaluations while maintaining consistent, reliable comparisons.

The Evidence

Experiments on two settings—online text classification and Terminal-Bench 2.1 (a benchmark for tool use)—show Task-CoEvolve consistently beats baselines that either select tasks randomly or use static heuristics. Crucially, it matches the final harness quality of exhaustive full-set search while requiring only 20% as many evaluations during the optimization process. This is a substantial practical win: optimization becomes faster without sacrificing the solution quality.

The work assumes that disagreement among candidate harnesses is a good proxy for informativeness, which holds in the tested domains but may not generalize universally. Still, the simplicity and effectiveness of the approach—no need to pretrain a task difficulty predictor or maintain auxiliary models—makes it a practical contribution for practitioners optimizing harnesses at scale.