From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

The Problem with Current AI Pentesting Benchmarks

Existing benchmarks for AI pentesting agents are too narrow. They measure success through artificial metrics—capture-the-flag completion, reproducing known exploits, or matching expected trajectories—in simplified environments. Real pentesting is messier: attackers must explore unknown systems, discover unexpected vulnerabilities across multiple attack surfaces, and make strategic decisions with incomplete information. Current evaluation doesn't capture this complexity, so benchmarks poorly predict which AI agents will actually work against real targets.

A Shift From Task Completion to Vulnerability Discovery

The authors propose a new evaluation protocol that measures what actually matters: whether an AI agent finds real vulnerabilities. Instead of checking if it completes a predefined goal, the protocol asks: did it discover exploitable security flaws that an expert would validate?

The method combines several techniques to handle real-world messiness. Structured ground-truth comes from expert annotations—known vulnerabilities in test systems. LLM-based semantic matching bridges the gap between how an agent reports findings and how vulnerabilities are formally documented, avoiding penalizing agents for valid discoveries phrased differently. Bipartite resolution handles cases where multiple findings map to the same vulnerability or vice versa, scoring results fairly under realistic ambiguity.

Because AI agents are stochastic (they produce different outputs each run), the protocol runs repeated evaluations and accumulates results over time, capturing both consistency and discovery breadth. Efficiency metrics track time-to-first-discovery and resource consumption, acknowledging that speed matters operationally. Finally, a reduced-suite selection mechanism allows researchers to run smaller representative subsets of tests for faster iteration without sacrificing fidelity.

Why This Matters

The shift from task-specific metrics to vulnerability discovery is operationally important. A pentester doesn't pass or fail based on whether it completed a predetermined sequence—it succeeds by finding exploitable weaknesses. This protocol aligns evaluation with real requirements, making benchmark results more predictive of field performance.

The authors release their ground truth annotations and code (ethibench) to enable reproducibility and standardization. This infrastructure lets the community evaluate agents consistently and compare results meaningfully, rather than each group using their own ad-hoc criteria.

By spanning multiple attack surfaces and vulnerability classes in complex targets, the protocol captures the strategic breadth real pentesting demands. Agents must learn not just to execute specific exploits but to navigate uncertainty, prioritize reconnaissance, and recognize diverse vulnerability patterns—precisely the skills that matter when deployed against unknown infrastructure.

The protocol doesn't claim to be perfect; real-world pentesting remains far more complex than any benchmark. But it moves evaluation closer to operational reality than current alternatives, offering a more credible answer to the question: which AI pentesting agent will actually find vulnerabilities in your systems?