An End-to-End Agent Auditing Engine

Current LLM-based agents rely on harnesses—software frameworks that orchestrate how models interact with tools and APIs. But evaluating these harnesses has been a mess: there's no standard way to test them, and most evaluation focuses only on whether an agent gets the right answer, ignoring how it gets there. The authors introduce A²E (Agent Auditing Engine), a systematic evaluation platform that reveals harnesses differ dramatically in execution efficiency, tool use, planning, and error recovery—and no single harness-model pair wins across all task types.

How It Works

A²E operates in three stages. First, task integration: the authors created a new Agent Task Protocol (ATP) that standardizes how evaluation tasks are plugged into different harnesses. This is the unsexy but critical piece—without it, comparing harnesses meant building custom adapters for each one.

Second, execution monitoring: during task execution, A²E's automated Monitor instruments the agent's behavior and generates standardized traces. Rather than just recording pass/fail, it logs every tool call, planning decision, and error the agent encounters. This creates a detailed record of how the agent solved (or failed) each task.

Third, multidimensional evaluation: instead of a binary correctness metric, A²E assesses harnesses across multiple dimensions. The paper doesn't specify all metrics in the abstract, but hints at: correctness (did it work?), execution efficiency (how many steps?), tool use quality (did it call the right tools?), task planning (was the approach sound?), and error recovery (how did it handle failures?). Each dimension captures something different about harness capability that a single accuracy score would hide.

Why It Matters

The experiments revealed a critical finding: model-harness combinations show substantial performance variation depending on task type, and no single combination dominates everywhere. This is important because it means practitioners can't just pick one "best" harness and call it a day. Instead, they need systematic evaluation to match harnesses to specific use cases.

The work also enables the "co-evolution" of models and harnesses mentioned in the abstract—teams can now see exactly where a harness is weak (poor error recovery, inefficient planning, excessive tool calls) and fix it, rather than guessing. It's the difference between knowing your car runs slow and knowing whether the problem is the engine, transmission, or aerodynamics.

Methodologically, A²E standardizes a currently fragmented landscape. As harness frameworks proliferate and evolve, having a common evaluation protocol prevents each research group from reinventing benchmarking wheels. The code release amplifies this: others can now rapidly add new harnesses or tasks to the framework without rewriting monitoring and metric code.

The real value is in the fine-grained diagnostics. An agent that answers 80% of questions correctly could be failing because it can't plan tasks properly, or because it panics on errors, or because it wastes steps on redundant tool calls. A²E separates those concerns, letting teams debug agent behavior systematically rather than stare at aggregate scores.