Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

The Problem: Failures Hide Their Origins

When an agent fails—missing a step in code generation, ignoring a user constraint, or choosing the wrong tool—teams can't tell where to fix it. Is the language model underperforming? Did the prompt (the "harness") poorly specify the task? Is the environment missing something? Is the benchmark itself flawed? Without this diagnosis, organizations waste effort applying the wrong fix. A single visible failure might demand model retraining, prompt engineering, tool redesign, or benchmark revision. The authors call this the "repair-assignment problem," and existing failure taxonomies don't solve it because they're tied to specific benchmarks and lack a shared language.

The Solution: Map Failures to Interactions

The authors propose an interaction-centric taxonomy that treats agents as systems composed of interacting parts: models, prompts/instructions (harnesses), users, tools, memory, and environments. Every failure originates at an interaction edge—between two components—and has a fault side indicating which component should be repaired.

For example:

• A model misunderstanding a user request lives on the user→model edge, with fault on the model side (post-training needed).
• A harness that doesn't clearly specify tool constraints lives on the harness→tool edge, with fault on the harness side (rewrite the prompt or scaffold differently).
• An evaluation metric that rewards incomplete solutions lives on the environment→grader edge, with fault on the environment side (fix the benchmark).

The taxonomy organizes 41 failure modes across these interaction edges. Crucially, a single failure mode maps to exactly one edge and fault side, making the repair target unambiguous.

Why This Matters in Practice

The taxonomy is grounded in real examples: public benchmarks like HumanEval and Spider, model system cards, published failure reports, and logged agent trajectories from coding assistants to multi-agent systems. This breadth shows the schema isn't benchmark-specific and transfers across different agent architectures.

To validate reproducibility, the authors used reasoning agents (themselves models with access to the taxonomy definition) as judges, asking them to categorize failures independently. Across four frontier models, the strongest judge achieved Cohen's κ = 0.76 against human labels—a substantial agreement level suggesting the categories capture real structure rather than subjective annotator preference.

Impact

This taxonomy shifts failure analysis from "what went wrong?" to "what went wrong where, and who fixes it?" Model teams can now pinpoint whether a failure indicates a training gap or a prompt design issue. Harness teams can distinguish between tool-integration problems and user-specification problems. Benchmark maintainers can identify evaluation flaws. The taxonomy is actionable: each category points directly to a repair strategy.

By making failure diagnosis systematic and shareable across teams and organizations, the work addresses a fundamental bottleneck in agent development—not just understanding failures, but quickly assigning them to the right team for the right fix.