ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

The Problem: Agent Benchmarks Miss What Actually Matters

Current AI agent benchmarks score only the final answer—whether the agent got it right or wrong. But agents don't work in isolation. They run on stateful systems (called runtimes) with tools, memory, messaging, scheduling, and the ability to delegate to subagents. When something fails, the failure might not be a wrong answer; it could be a mistake in how the agent navigated the runtime, violated a safety rule, or struggled with repeated execution. Measuring only the final result hides these failures and gives an incomplete picture of what a model-plus-runtime system can actually do.

How ClawProBench Works

The authors built ClawProBench on OpenClaw, a live agent runtime with workspace tools (browsing, memory, messaging, scheduling, skills, subagents) that mirrors real-world setups. The benchmark has two tracks:

Full profile: 102 scenarios where agents interact with live workspaces and native-runtime routing tasks—the messy, real-world version. This uncovers how agents actually fail in production.

Frozen holdout: 68 closed-world scenarios with fixed JSON output contracts, designed for stable ranking and comparison over time. This ensures robust leaderboard positioning.

Scoring comes from execution traces (records of everything the agent did), not just final outputs. A safety-gated formula combines three dimensions: correctness (did it work?), process quality (did it work well?), and efficiency (did it work fast?). Crucially, failure evidence is preserved in traces for audit—you can see where and why things went wrong.

Key Findings: Hidden Weaknesses in Final-Answer Leaderboards

Evaluating 68 configurations on the full profile and 37 on holdout revealed striking gaps in naive leaderboards:

• Native-runtime tasks are harder: Tasks using native surfaces scored 0.5238 versus 0.6415 for live workspace tasks. Final-answer scoring would miss this systemic weakness.
• Rankings don't transfer: Full-profile and holdout rankings showed weak agreement (Spearman ρ = 0.1300). A model ranked high on the full benchmark might rank lower on holdout—suggesting final-answer metrics hide one-off successes.
• Different scoring tells different stories: Rankings based on correctness alone differ substantially from rankings that account for process quality, safety gates, and strict trial limits. A model might pass once out of three attempts (pass@k-any: 0.6638) but never consistently (strict pass: 0.2890).

The top safety-gated score across all configurations was 0.7671, showing room for improvement across the board.

Why This Matters

Agent systems are moving into mission-critical domains where not just the answer, but how the agent got there, matters for safety, compliance, and reliability. ClawProBench forces visibility into runtime behavior, failure modes, and process quality—dimensions that final-answer leaderboards erase. By preserving traces and scoring process alongside correctness, the benchmark creates accountability for the full model-runtime configuration, not just luck on the final output.