- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 6
A conversation between
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
§02
Snippets
-
Agent Retrieval Bench isolates file-level context retrieval as a distinct evaluation problem, measuring what agents actually need rather than semantic query-file similarity.
Patch correctness depends on retrieval success, yet most benchmarks skip this upstream stage, hiding a critical failure mode.
-
Relevance is defined by agent workflow signals—what a developer actually consulted next—not static semantic similarity.
This grounds evaluation in real task context rather than abstract relatedness, catching mismatches between ranking and usefulness.
-
A selective-abstention subset uses natural no-gold cases and counterfactual wrong-repository controls to test whether systems confidently refuse impossible queries.
Prevents systems from wasting agent context on searches with no correct answer—a silent failure in retrieval pipelines.
-
Qwen3-Embedding-4B ranks highest on precision, Qwen3-Embedding-8B on recall, and RepoMap on token-budgeted yield; winners vary by task.
Method choice matters more than general-purpose ranking, suggesting task-adaptive or ensemble strategies may be needed.
-
Thresholds learned from counterfactual controls fail to improve selective success on natural no-gold cases, revealing a calibration gap.
Selective abstention is harder than it looks; counterfactual data doesn't capture real distribution drift in live agent usage.
-
Retrieval-derived context yields higher file F1 than random non-gold context, with less post-seed exploration needed.
Good retrieval reduces exploration overhead, suggesting it's a genuine bottleneck upstream of patch generation.
§03
Synthesis
The Missing Piece in Coding Agent Evaluation
Coding agents are typically judged on whether they produce correct code patches, but that success depends entirely on an earlier, overlooked step: finding the right files in a repository to work with. This paper argues that file retrieval is a distinct and hard problem that deserves its own benchmark. The authors find that no single retrieval method dominates across tasks, selective abstention (deciding when to say "I don't know") fails to generalize from controlled test data to real scenarios, and even human-logged agent trajectories miss relevant files in 27–35% of cases.
How the Benchmark Works
Agent Retrieval Bench contains 427 samples built from real coding workflows across 25 open-source repositories. Rather than asking "which files are semantically similar to the query?", the benchmark defines relevance as "which files does an agent actually need to solve the task next?" This is a crucial distinction. The authors derive ground-truth file needs from four practical scenarios:
- code2test: given source code, find test files that exercise it
- comment2context: given a code comment, find files it references
- trace2code: given an execution trace, find the source files involved
- edit2ripple: given an edited file, find files affected by that change
A fifth subset tests selective retrieval: can methods correctly say "no relevant files exist" in natural cases and resist false positives on counterfactual wrong-repository controls?
The corpus spans 308 repository snapshots and 7.9 million code chunks. Retrieval quality is measured by Mean Reciprocal Rank (MRR) on positive examples and F1 scores on file-level predictions.
Key Findings and Why They Matter
The benchmark reveals fragmentation across retrieval families. Qwen3-Embedding-4B achieves the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B wins on Recall@20 (finding at least one correct file in the top 20), and RepoMap—a graph-based method—yields the best contextualization at fixed token budgets (8K tokens). No method excels universally, and task winners differ substantially. This suggests that repository structure, code semantics, and workflow context interact in ways that favor different approaches depending on the problem.
More concerning: selective abstention fails in practice. The authors calibrated confidence thresholds using counterfactual controls (swapped-in irrelevant repositories), but these thresholds did not improve decisions on natural no-gold cases where no truly relevant files exist. This calibration gap hints at a distribution mismatch between test controls and real ambiguity.
Finally, a controlled seed-intervention experiment shows that retrieval-derived context (files ranked by a retrieval model) yields higher file-level F1 and less subsequent exploration than random non-gold context—but oracle gold context still shows substantial headroom, meaning even perfect retrieval would not solve the full problem.
Why This Matters
Coding agents' success is bottlenecked by context acquisition. If an agent never sees the right file, no patch generation method can fix it. This benchmark isolates that upstream problem, providing a foundation for improving retrieval and revealing the real limitations of existing methods and evaluation practices.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.