Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

The Problem: Multiple-Choice Tests Hide What LLMs Actually Know

Standard multiple-choice benchmarks for evaluating large language models conflate two distinct things: whether a model knows the answer and whether it's swayed by where the correct answer sits in the list. A model might score 75% on a test but only because it has learned to pick option B more often than C—a spurious signal that inflates confidence in its knowledge. The authors ask whether removing this positional bias through "label-free" strategies can both eliminate order sensitivity and boost accuracy.

Two Label-Free Approaches Fall Short

The researchers test two strategies designed to hide option labels until the model has committed to an answer:

Generation-then-matching. The model generates its own answer first (without seeing the options), then an LLM matcher maps that answer to one of the multiple-choice options. This decouples knowledge from option order.

Scoring in isolation. Each option is scored independently, with no positional context, making the method "unbiased by construction." The model evaluates each choice separately and the highest-scoring option wins.

Neither strategy reliably improves accuracy over a baseline where the model simply reads all options and picks one. The critical bottleneck, the authors discover through complete decomposition, isn't the matching or scoring step—it's the fact that withholding options during generation severely constrains what the model can see and reason about.

Order Sensitivity ≠ Accuracy Improvement

Here's the counterintuitive finding: eliminating positional influence entirely does not reliably yield accuracy gains. The only configuration matching baseline performance requires showing the model all options paired with an LLM matcher—at which point positional bias remains intact.

Conversely, simply cycling through permutations of option order (e.g., testing with options in different sequences) often improves accuracy on its own, even without any debiasing strategy. This suggests that order sensitivity and accuracy improvements operate on different mechanisms. A model might be sensitive to position yet still benefit from seeing options presented multiple ways.

To measure whether debiasing actually worked, the authors applied two diagnostic metrics: an aggregate measure of "recall imbalance" (whether the model favors certain positions) and a per-question measure of order sensitivity. Both failed to show reliable debiasing across the tested strategies, further decoupling positional bias from performance gains.

Why It Matters

The core insight is that multiple-choice benchmarks are more muddled than previously assumed. Researchers often assume that reducing positional bias will automatically improve reliability and accuracy—but the two diverge. A strategy might eliminate order sensitivity without boosting knowledge detection, or vice versa. This means that simply making benchmarks "position-unbiased" is neither necessary nor sufficient for better evaluation.

For practitioners, the takeaway is sobering: fixing one problem (option-order influence) doesn't fix the other (measurement of true capability). Benchmarking LLMs remains tricky, and label-free strategies offer no silver bullet.