The Loss Does Not See the Basis, but Adam Does

The Bias Hiding in Plain Sight

Gradient descent and Adam start from the same place but end up making different choices about what solution to find. When fitting a factored model (decomposing a weight matrix W into a product UV^top), gradient descent naturally gravitates toward low-rank solutions—sparse in a particular sense. Adam, the workhorse optimizer in modern deep learning, does not. The authors' central finding: this divergence is not about optimization speed or convergence rate, but about what the optimizer is fundamentally allowed to see.

The culprit is gauge symmetry. The loss function treats (U, V) and (UQ, VQ) as identical for any invertible matrix Q—rotating both factors together doesn't change the output. This symmetry is invisible to the loss itself, but it's not invisible to all optimizers. Gradient descent respects this symmetry (it's "gauge-equivariant"); Adam breaks it by treating each coordinate independently, destroying the symmetry at the first step. Once broken, Adam locks into a particular choice of basis and can no longer drift toward low-rank solutions that gradient descent finds naturally.

Which Optimizers See the Symmetry?

The authors prove a structure theorem: memoryless optimizers that respect gauge symmetry are exactly those using Gram-determined left preconditioning—a technical characterization, but the practical takeaway is clear. Gradient descent, momentum methods, and "shared-scalar" variants of Adam (where a single scalar controls the step size across all coordinates) preserve the symmetry. Coordinate-wise methods—standard Adam, RMSProp—destroy it immediately. Muon and Shampoo sit on the boundary, depending on their spectral schedule.

On matrix sensing tasks (recovering a low-rank matrix from noisy observations), the authors sort nine update rules by how well they recover ground truth. A one-parameter family interpolating from coordinate-wise to shared-scalar preconditioning restores low-rank bias monotonically, confirming that anisotropy—treating different directions differently—is the mechanism that breaks the symmetry.

Real Consequences

The impact shows up in transformers and on hyperspectral imaging. In transformers, Adam separates two gauge-equivalent initializations by 56% relative Frobenius distance at the first step, a gap no per-head rotation can repair. On hyperspectral data at low sampling density, gradient descent cuts error by 43–44% compared to Adam, achieving lower effective rank. These are not marginal differences.

The authors argue this reframes a common assumption: basis choice is often treated as a tuning detail, a freedom that doesn't matter. Instead, it's a fundamental decision about which interpolant—which solution among many equally valid ones—the optimizer selects. Adam implicitly commits to one basis; gradient descent drifts toward solutions that remain invariant across basis choices. Neither is universally "better," but their inductive biases are opposite and now explicit.

The paper's significance lies in rendering invisible geometric structure visible: showing that optimizer design encodes hidden assumptions about what solutions are worth finding.