Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking

The Problem: Reasoning Models That Talk Themselves in Circles

Large language models trained to "think out loud" through multi-step reasoning often sabotage themselves. They generate lengthy chains of hedging, abandoned approaches, and self-contradictions—behaviors that burn tokens without improving final answers. The surprise: this overthinking isn't just about length. Even when two reasoning traces are the same length, the incorrect ones contain more of these unproductive self-reflection patterns. The authors' central insight is that we can identify and penalize overthinking without expensive human annotations of every reasoning step.

The Solution: Learning from Answer Checkpoints

The key innovation is elegant: reasoning traces often contain intermediate answer commitments—points where the model proposes a candidate solution. These free checkpoints act as a signal for whether subsequent reasoning helps or hurts.

Here's how it works. Suppose a model generates a reasoning chain that arrives at answer A, then reconsiders and pivots to answer B. The authors compare both A and B to the ground truth. If A was already correct and the model switched to an incorrect B, that's a red flag: the subsequent reflection was counterproductive. If A was wrong but B is correct, the reflection added value. No human needs to annotate the intermediate reasoning—only compare answers to labels already available during training.

DASH (Drift Aware advantage SHaping) operationalizes this by assigning credit at the segment level—grouping reasoning steps between answer commits and rewarding or penalizing them based on whether the final answer after that segment moved closer to or further from correctness. This is notably different from token-level or step-level credit assignment, which would require fine-grained supervision.

Why It Matters

On competition-level math benchmarks (where overthinking is most prevalent), DASH reaches 59.45% accuracy versus 58.1% for Dr.GRPO and 56.95% for standard GRPO. More important than the margin: the model simultaneously reduces overthinking behaviors and performs better self-correction. The method also scales—it requires no new human labels, only existing problem-answer pairs.

The work tackles a real practical problem. Reasoning models are now commonly used for math, code, and planning tasks where extended thinking is valuable. But longer reasoning doesn't automatically mean better reasoning. DASH offers a way to teach models to self-edit productively: to know when reflection helps and when it's just repetitive doubt. The intermediate-answer-checkpoint signal is both cheap and surprisingly informative, making the approach practical for teams fine-tuning reasoning models with modest budgets.