Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression

The Problem: Compression That Breaks Logic

Hard prompt compression—keeping only the most important tokens or sentences from long documents—creates a hidden structural failure. When a compressor independently rates each piece of text, it can accidentally split apart dependent information. Imagine a document where sentence A contains an answer ("The capital is Paris") but only sentence B defines what "it" refers to ("France is a country"). A greedy compressor might keep A but discard B, leaving the answer floating without context. The authors call this referential dangling.

This isn't a minor edge case. At a compression ratio of 0.30 (keeping 30% of tokens), Beaver—a chunk-ranking system using Qwen embeddings—introduces dangling references in 34–54% of multi-hop reasoning examples. Testing six popular hard compressors on HotpotQA, the authors found dangling rates up to 60%, and every single document in LongBench's QA benchmark contained at least one dangling reference.

Measuring the Cost

The damage is real. When the authors manually reinsert the missing supporting paragraph (while removing irrelevant ones to stay within the token budget), accuracy improves by 29–34 percentage points on HotpotQA and MuSiQue—recovering 88% of the gap compared to keeping both paragraphs. Crucially, stronger answer models don't fix this: GPT-4o is still 8.8 points less accurate on compressed contexts with dangling than on complete ones.

The Fix

Rather than redesigning compression from scratch, the authors train a lightweight classifier to detect when a retained sentence needs support from deleted text. At inference time, this classifier ranks omitted sentences by whether they're needed to interpret what's kept, then reinserts the top candidates—no annotation required at test time.

On HotpotQA with Qwen-8B, this approach recovers 4.7 accuracy points while barely changing the compression ratio (0.30 → 0.31). The key insight: you don't need to find all missing context—just the most critical gaps.

Why It Matters

This work exposes a paradigm-level flaw in how modern systems compress long contexts. Relevance scoring is necessary but insufficient; a sentence can be "irrelevant" in isolation yet essential for interpreting an answer elsewhere. The fix is practical—a small trained module—and the problem is widespread enough that it should reshape how practitioners think about compression. Rather than asking "Is this sentence important?", they should ask "Is this sentence important given what we're keeping?"