- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 4
A conversation between
The Laws of Context Allocation: Causal Measurement and Closed-Loop Orchestration in Generative Search
§02
Snippets
-
Standard relevance proxies fail catastrophically on hard negatives; a causal leave-one-out probe accurately isolates what evidence the model truly relies on.
Measuring actual generative reliance, not just surface relevance, is essential to optimize context allocation in RAG systems.
-
Monolithic context widening is an architectural trap penalized by relevance decay; sequential multi-generation allocation across smaller contexts drives 16.7–20.5 percentage point gains in portfolio recall.
Iterative allocation scales better than bigger single passes, reshaping how search systems should structure context use.
-
An attribution-steered contrastive decoder overrides LLM attention inertia, forcing systematic integration of newly retrieved evidence across iterations.
Structural steering at decoding time ensures the model doesn't waste retrieved evidence by falling back on learned priors.
-
A closed-loop submodular scheduler jointly optimizes sequential retrieval and generation decisions, embedding causal measurement into live allocation feedback.
Treating context allocation as a dynamic feedback loop rather than a fixed pipeline unlocks systematic gains across model scales up to 32B.
§03
Synthesis
The Problem: Broken Feedback in RAG Systems
Retrieval-Augmented Generation (RAG)—where language models pull in external documents to ground their answers—currently suffers from two hidden failures. First, nobody knows whether the model is actually using the retrieved evidence; standard relevance metrics (like matching query to document) collapse on adversarial cases. Second, systems naively widen context windows, but this plateaus and often backfires. The authors show these aren't separate problems—they're symptoms of flying blind.
Measuring What the Model Actually Uses
The core insight is a "diagnostic illusion": when you ask whether a retrieved document is relevant, you're measuring the wrong thing. A document can rank high on standard metrics yet the language model ignores it, especially when similar-looking but wrong documents (hard negatives) are nearby.
The authors replace this with a causal leave-one-out probe. The idea is simple: remove one piece of evidence and measure how much the model's output changes. If removing a document shifts the generation meaningfully, the model relied on it. If removing it changes nothing, the model didn't use it. They formalize this as isolating "generative reliance" and quantify how much the model's attention gets diluted across multiple sources—a structural property of attention, not a surface relevance score.
Rethinking Context Allocation
With accurate measurement in hand, the authors tested whether dumping more documents into the context window helps. It doesn't—at least not monotonically. Instead, they propose sequential allocation: generate an answer using some evidence, measure what actually helped via the causal probe, then iteratively feed fresh evidence in a second or third pass.
This "closed-loop" strategy yields major gains: 16.7–20.5 absolute percentage points on portfolio recall (how many correct answers across multiple queries). Monolithic widening—the industry standard—is an "architectural trap." Iterative, feedback-driven allocation works better and scales to 32B parameter models.
The Deployed System
The authors package this into a closed-loop submodular scheduler: a controller that decides which evidence to feed in which generation round, guided by the causal measurement. They add an "attribution-steered contrastive decoder" that nudges the model to use new evidence rather than relying on attention patterns from earlier passes.
Why It Matters
Most RAG systems are black boxes: you throw in documents and hope. This work opens them up. The causal measurement exposes when evidence is wasted (a pervasive waste today), and the sequential allocation strategy flips the scaling paradigm—more is not always better; better feedback is. For deployed search systems handling millions of queries, cutting context waste while improving accuracy is a direct win on latency and cost.
The authors promise code and measurements open-source, making this reproducible and adoptable. The shift from static "throw everything in" to adaptive "feed evidence iteratively based on what works" is a conceptual reframing for generative search.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.