Maglev: Sliding Recurrent Memory

The Core Innovation

Maglev solves a fundamental tension in language models: how to maintain long-range context awareness while keeping memory footprint bounded and training parallelizable. The authors' key insight is that you don't need one model doing both jobs poorly—use two coupled models instead. A "prefiller" Q sees the full history and distills it into compact memory targets. A "decoder" P learns to approximate that compressed memory using only a sliding window (recent tokens) and recurrent updates, enabling both efficient inference and fast training.

How It Works

During training, Maglev operates in two stages. The prefiller Q processes the entire sequence with full or interleaved attention, generating ideal memory representations m'_t at each step. Simultaneously, the decoder P works with a restricted sliding window—attending only to the last N tokens—but also receives injected K/V (key-value) pairs from its own previous memory state. This recurrent injection lets P "remember" older information without attending to it directly.

The two models are trained jointly with a memory consistency loss that penalizes misalignment between Q's targets m'_t and P's actual memories m_t. Over time, P learns to approximate Q's superior memory using only local attention and recurrence. At inference, you discard Q entirely and run P alone—it's small, memory-efficient, and fully parallelizable.

A practical detail: Q uses interleaved full and sliding-window attention rather than pure full attention, which improves performance while maintaining tractability.

Why This Matters

Traditional sliding-window attention (e.g., in Mamba or some Transformer variants) is fast and memory-bounded but loses long-range dependency modeling. Latent recurrent transformers keep history but either break parallelism or require expensive state updates. Maglev threads the needle: it recovers long-context understanding during training via the prefiller, then distills that capability into a recurrent model that's cheap at inference time.

The empirical results validate this trade-off. Maglev improves validation loss and downstream pretraining benchmarks compared to both sliding-window baselines and latent recurrent transformer competitors. Notably, the authors show that sharing parameters between P and Q (reducing parameter overhead) preserves most gains, making the approach practical for large-scale training.

The parallelizability during training is crucial—unlike purely recurrent models, Maglev doesn't force sequential processing of sequences during learning, so it scales to realistic dataset sizes. This combination of training speed, inference efficiency, and improved perplexity positions Maglev as a compelling alternative to standard attention for sequence modeling, especially in settings where context length matters but computational budget is constrained.