Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

The Problem: Benchmark Contamination Looks Better Than It Is

When AI models train on internet-scale data, they inevitably memorize chunks of popular benchmarks. This inflates evaluation scores—a model might "solve" a question not because it understands the task, but because it regurgitated cached training data. Researchers have developed mitigation strategies: techniques that intervene during generation to suppress memorization and reveal the model's true capability. But there's a measurement crisis. The dominant metric, G-AP (Gap of Aggregate Performance), is fundamentally broken, and nobody has noticed how badly.

The authors expose three concrete flaws. First, G-AP bins answers as correct/incorrect, throwing away confidence information. A model that assigns 52% probability to the right answer looks identical to one assigning 99% probability—both read as "correct." Second, G-AP averages first, then computes the difference. If a mitigation strategy over-suppresses on some questions and under-suppresses on others, those errors cancel out, masking poor performance. Third, the metric weights all questions equally, which creates a perverse incentive: a strategy can artificially lower its contaminated score by pushing answer probabilities toward whatever the clean (uncontaminated) model frequently gets right, even if the underlying capability hasn't improved.

The Solution: Stratified Per-Question Probability Gaps

The authors propose SA-PPG, which flips the evaluation logic. Instead of binaries, it estimates each question's solve probability by sampling multiple times and counting successes. For each question, it computes the probability gap—how much the contaminated model's solve probability drops after mitigation—then compares that against what the clean model would achieve. Crucially, it aggregates these gaps in stratified buckets: questions are grouped by how often the clean model solves them (easy, medium, hard), and gaps are averaged within groups. This prevents low-difficulty questions from drowning out high-difficulty ones, and forces honest accounting of suppression across the difficulty spectrum.

The authors then introduce RailCap, a mitigation strategy that operates during generation rather than by first detecting contamination. Whenever the model's greedy prediction (the single most likely next token) matches its previous trajectory, RailCap caps that token to the runner-up option instead. This forces diversity in the output distribution, accumulating suppression until the model's responses become sufficiently dispersed—closer to what an un-memorizing model would produce.

Why It Matters

Across multiple contaminated models and benchmarks, SA-PPG reveals that prior mitigation strategies—ones claimed to "restore" genuine capability—are substantially overestimating their success. The new metric exposes strategies that looked good under G-AP but actually fail to suppress memorization in meaningful ways. RailCap, by contrast, achieves the lowest SA-PPG scores, suggesting genuine restoration.

This is not a minor fix. If evaluation metrics systematically overstate mitigation success, the field has been chasing inflated progress. SA-PPG and RailCap together provide both honest diagnosis and a stronger intervention, moving benchmark evaluation from false confidence toward genuine capability measurement.