When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

Modern language models often solve hard problems by generating many candidate answers and picking the best one. Intuitively, more samples should mean more chances to find a correct solution. This paper challenges that assumption: beyond a modest number of samples (often dozens), additional attempts produce no measurable gain and can actively harm final performance, even though they contain correct answers the system cannot identify.

The Core Problem: Coverage vs. Selection

The authors identify a critical gap between two metrics. Coverage — the fraction of problems where at least one sample is correct — climbs steadily with more samples. A deployed system, however, must return a single answer. Selection — picking the right answer when you don't know which samples are correct — plateaus much earlier. Voting or confidence-based ranking settles quickly; after that, more samples only reinforce confident mistakes.

The authors call this gap the identifiability gap: correct answers exist in later samples, but the selection mechanism cannot reliably recognize them. The real bottleneck is not generating candidate answers but identifying which one is right.

Two Concrete Ceilings

The paper introduces two practical limits:

Modal ceiling: When majority voting is used to pick an answer, consensus typically stabilizes within a few dozen samples. Adding more votes after this point changes the winner rarely, if ever. The vote distribution has essentially locked in.

Correlation ceiling: When benchmarking a system's overall performance (e.g., pass@k metrics on a test set), the correlation between sample counts stops improving much sooner — often after just 10–20 samples per problem. Extra samples add cost but don't shift which problems the system solves correctly at the aggregate level.

Effective Number of Samples

Rather than fixing a sample budget in advance, the authors propose computing an effective number of samples from any existing run. This single number reveals when diminishing returns have set in, allowing practitioners to recognize when they should stop sampling. The metric directly estimates where the ceiling has already been hit.

Why This Matters

Test-time scaling — sampling more at inference time rather than training longer — has become a key strategy for improving reasoning models. But it is expensive. The paper's core insight reframes the question: it is not whether sampling helps (it does, up to a point) but where that point lies, and whether stopping there leaves performance on the table.

The authors show that recognition, not generation, is the limiting factor. A model can produce correct answers it will never select. This shifts focus from "sample more" to "select better" — suggesting that research should prioritize improving confidence estimation, verification methods, or other selection mechanisms rather than scaling up sample counts indefinitely.

For practitioners, the result is a blunt warning: blindly increasing samples to boost metrics can backfire. The paper provides a way to spot when that threshold has been crossed.