Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Cost changes everything in security agents—and no one was measuring it

Current benchmarks for AI security agents are broken. They measure whether an agent can find a vulnerability or solve a CTF challenge given unlimited compute, but ignore a critical reality: in production, every action costs money. The authors evaluated offensive and defensive security agents through a cost-success lens and found that red-team and blue-team tasks scale completely differently—a finding hidden by traditional success-rate reporting.

The problem with "success at any cost"

Security evaluations typically report peak performance: did the model solve the challenge? This ignores operational constraints. In real security operations, every reasoning step, API call to a forensic tool, and data enrichment request consumes budget. A model that solves 95% of challenges using $100 of compute is useless if a competitor solves 90% for $10.

The authors tested two domains: Cybench (offensive red-team challenges like CTF competitions and penetration testing) and Splunk BOTS v1 (defensive blue-team challenges simulating SOC (Security Operations Center) investigations). For each task, they measured both success rate and the cost breakdown—separating reasoning compute (inference) spend from tool-call spend.

Red-team and blue-team scale differently

Offensive agents show classic scaling: more test-time compute means better performance. Scaled open-weight models (like larger versions of Llama) can approach proprietary frontier systems (like Claude or GPT-4) while remaining cost-competitive. Additional reasoning steps help agents explore attack paths, refine exploits, and retry failed attempts. This is good news for cost-conscious operators.

Defensive agents tell a different story. Performance does not scale primarily with reasoning budget. Instead, success depends on disciplined tool use—knowing which forensic query to run, interpreting telemetry efficiently, and avoiding wasteful enrichment requests. A defender with excellent reasoning but poor tool discipline will burn budget quickly and solve fewer incidents. This suggests that blue-team agents need different training and evaluation criteria than red-team agents.

Why this matters

The findings have two implications. First, existing benchmarks underestimate the practical utility of open-weight models in offensive scenarios; scaled models can compete at lower cost. Second, the defensive results expose a blind spot: current evaluation metrics miss critical inefficiencies. A model might hit 80% success on Splunk BOTS in the lab with generous budgets, but fail in production where telemetry is expensive and SOC analysts demand rapid triage.

The authors argue that security-agent benchmarks should report cost-success trade-offs, not just peak success. They provide an interactive website with disaggregated results (inference cost vs. tool cost) to let practitioners make informed model choices for their threat model and budget.

This reframes how teams should evaluate and deploy security agents: not "which model is smartest?" but "which model solves my problem at a cost I can afford?"