StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

The Core Finding

Long-horizon AI agents fail not because their language models lack capability, but because they lose track of what they're doing. StateM—a runtime system that wraps around an agent without retraining it—pushes GPT-5.6 Sol Ultra to 95.3% accuracy on Terminal-Bench 2.1, a significant jump from the 83.1% baseline. The same system cuts costs dramatically: DeepSeek-V4 Flash reaches 89.1% accuracy for $52.22 total API spend, versus $574.68 for the reference GPT approach.

How It Works

Instead of betting on bigger or better models, the authors built an execution harness—think of it as a smart wrapper around the agent that keeps things organized. StateM introduces five key mechanisms:

Durable states anchor the agent's memory. Rather than letting the agent hallucinate or forget what phase of a task it's in, StateM maintains explicit, checkpointable state that persists across steps.

Phase-local context means the agent only sees the relevant information for its current step, cutting noise and confusion.

Checked transitions enforce rules about what actions are legal next. If the agent tries an invalid move, the harness catches it before execution, not after failure.

Recoverable runbooks are executable procedures—sequences of steps the agent has learned work for common patterns. If a runbook fails partway, the system can restart from a checkpoint rather than abandoning the whole attempt.

Versioned procedural practices let agents and humans inspect and refine what works. Successful strategies are recorded, versioned, and reused across similar tasks.

Critically, the model weights never change. The intelligence comes from organizing execution, not retraining.

Why It Matters

The results expose a gap between raw model capability and agent reliability. GPT-5.6 Sol can likely handle the individual steps of Terminal-Bench tasks—the benchmark tests Linux command sequences, file operations, and multi-step reasoning. But in a 10- or 20-step pipeline, small errors compound: the agent forgets which directory it's in, skips a necessary setup step, or stops early thinking it's done.

StateM recovers that lost performance cheaply. On BusinessBench (a corporate workflow benchmark), family-specific runbooks built from development data yield held-out gains of 0.55 to 1.34 points on unseen tasks. Two mechanism-matched families improve by 10 points, showing that the rules transfer meaningfully.

The cost angle is striking. Reaching 89.1% on a common core of Terminal-Bench tasks with DeepSeek-V4 Flash costs $52.22 total—about 1/10th the GPT reference bill. For cost-sensitive deployment, harness scaling offers a practical alternative to paying for frontier models.

The implicit claim: execution discipline, not model scale, is the bottleneck for many agent tasks. State management, phase isolation, and learnable runbooks are cheaper and often more reliable than chasing the next model checkpoint.