Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

The Puzzle: Why Do Sparse Experts Work If They're Not Geometrically Distinct?

Sparse mixture-of-experts (MoE) models—like Mixtral and DeepSeek—route each token to only a handful of experts rather than using all of them. The prevailing explanation is geometric: co-selected experts should push representations in different directions, each filling a unique gap. This paper challenges that story. The authors find that expert subspaces actually overlap substantially, yet routing still works. The question is: how?

The authors propose a new concept called coherent overlap. Rather than experts being geometrically distinct, they're coherently selected from a shared neighborhood—each chosen expert is relevant to the token's context, but they're not orthogonal to each other. Importantly, this overlap does not mean they're redundant; adding more experts still improves predictions.

How They Measured It

The authors disentangle three confounded quantities using controlled experiments across six MoE architectures (OLMoE, Mixtral, DeepSeek, and others).

Expert Subspace Separation Index (ESSI) quantifies geometric overlap directly. The finding: substantial overlap across all models—the geometric account alone doesn't explain routing success.

Matched-route residuals isolate whether the actual route choice matters. They compare how much a selected expert explains the residual representation (what's left after the first expert contributes) versus how much the strongest unselected expert would explain if swapped in. Across all 39 factorial cells tested in OLMoE, Mixtral, and DeepSeek, the selected expert always wins—but this advantage shrinks when you condition on the prefix (context). Every 95% confidence interval for the prefix effect is negative, meaning context interaction narrows the gap.

Frozen-route interventions test functional value: do later experts actually help prediction? They hold routing fixed and measure next-token accuracy with Top-1 versus Top-2 experts. In 24 of 39 cases, adding the second expert improves accuracy; the other 15 are inconclusive. A controlled training study confirms Top-2 outperforms Top-1 across all three seeds.

Why It Matters

This work breaks a myth. Prior work often treated geometric complementarity (disjoint directions) as necessary for MoE benefits. Here, the authors show that relevance-based selection from a shared space is sufficient—and that's what's actually happening.

The practical implication: pruning strategies that assume experts must be geometrically distant may over-compress models. A token-relevant expert from a crowded neighborhood can still contribute unique computation (perhaps through nonlinear interactions or later-layer mixing) even if its subspace overlaps with peers.

The methodological contribution is equally important. By separating route coherence, candidate quality, and context interaction via factorial design, the authors provide a template for untangling what drives routing decisions in sparse models. This clarifies why "similarity ≠ redundancy" and why geometric measures alone cannot determine which experts to prune or which to keep.