AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

The Problem With Current Code-Agent Benchmarks

Today's code-agent benchmarks collapse everything into pass/fail. Did the agent write correct code? Yes or no. But in real use, engineers experience far more: Does the agent follow instructions clearly? Does it verify its work? How does it recover when it makes mistakes? What does its reasoning look like? A single binary verdict tells you almost nothing about these dimensions.

AgentLens changes this by scoring the entire trajectory—the sequence of steps, tool calls, reasoning, and recovery attempts an agent makes—not just the final outcome. This shift matters because practitioners need to understand why a run succeeded or failed, and where their agent excels or stumbles.

How It Works

AgentLens combines three evaluation strategies:

Formal verification handles tasks with objective ground truth (e.g., "does this code compile and pass tests?"). These get a clean yes/no.

LLM-written trajectory reviews assess the harder parts: instruction-following, self-verification, error handling, and communication quality. An LLM reads the full agent trace—every action, thought, and tool call—and rates these dimensions. Each review includes a written explanation, not just a score.

Side-by-side comparisons let evaluators directly contrast two agent runs on the same task. This is particularly useful for detecting subtle differences between model versions.

The result for each run is a readable explanation of the score, not just a number. This makes the benchmark useful beyond leaderboards: the authors use it to diagnose why a model fails, compare their own agent versions, and catch performance regressions in nightly evaluation pipelines.

Why This Matters

Code agents are moving into production. Teams building these tools don't just want rankings—they want diagnostic insight. A developer might learn that one model fails because it doesn't verify its work, while another struggles with complex instructions. These insights drive concrete improvements.

The multi-faceted scoring also catches real issues that a pass/fail metric would miss. An agent might produce correct code through confused reasoning, or fail a task but demonstrate strong recovery behavior. Binary benchmarks are blind to these trade-offs.

AgentLens makes evaluation actionable in a way traditional benchmarks don't. By pairing objective checks where possible with LLM-based trajectory analysis and direct comparisons, it surfaces the patterns that actually matter to users. The open-source release at https://github.com/agent-lens/agent-lens-bench makes this accessible to the broader agent-building community, moving the field toward evaluation practices that reflect real-world deployment needs rather than abstract leaderboard dominance.