Memory for Large Language Models

The Fragmented Landscape of LLM Memory

Large language models have historically treated memory as an invisible consequence of their computations—information stored implicitly in attention weights and hidden states, accessible only during forward passes. The field has now splintered into dozens of competing approaches: some models add explicit retrieval mechanisms, others adopt recurrent loops, still others cache key-value pairs or learn task-specific adapters. This explosion of diversity has left researchers without a shared vocabulary or coherent mental model. The authors of this survey tackle that fragmentation head-on, proposing a unified architecture-centric taxonomy that organizes memory mechanisms along three fundamental dimensions.

Three Axes, One Framework

The authors characterize every memory approach in LLMs using three orthogonal properties. Representation distinguishes implicit memory (information baked into network weights or attention patterns) from explicit memory (separately stored data structures that the model can read and write). Update dynamics split between offline (memory set once, before inference) and online (memory modified during inference based on new inputs). Persistence separates short-term memory (available only within a single conversation or sequence) from long-term memory (retained across multiple interactions or deployments).

This framework lets researchers map existing systems—from transient attention mechanisms to parameter-efficient fine-tuning, from recurrent state dynamics to scalable lookup tables—onto the same coordinate system. A simple attention head is implicit, online, and short-term. A retrieval-augmented generation system using a fixed database is explicit, offline, and long-term. A continual learning system that updates its parameters mid-conversation becomes implicit, online, and long-term.

Bridging Separate Worlds

The critical insight is that memory operates along a spectrum between two extremes: computation-coupled (where storage is intertwined with the model's forward pass, like in standard transformer attention) and independently addressable (where data lives in a separate store and is fetched on demand, like in memory-augmented neural networks). Most current approaches cluster near one pole or the other, but the taxonomy reveals that hybrid architectures—mixing both styles—often emerge as practical solutions.

The authors dig deeper into the mechanisms that underpin these designs: how memory gets written (which tokens or states are stored?), how it routes (how does the model decide what to retrieve?), how state transitions occur (how does memory evolve with new data?), and how consolidation works (when does temporary memory become permanent?). Formalizing these operations turns intuitions about "memory" into precise computational patterns.

Why This Matters

The survey also addresses system-level concerns: computational cost, latency, storage footprint, and how different memory strategies trade off against one another. A model with massive explicit long-term storage may offer perfect context recall but slow inference; one relying entirely on parameter updates avoids retrieval overhead but struggles to adapt quickly to new information.

By consolidating scattered advancements into one coherent framework, the authors provide a roadmap for future work—showing where gaps remain, which combinations remain unexplored, and how to design memory systems that scale with both model size and real-world demands. In a field moving at breakneck speed, this unification offers clarity on what memory is and what it could become.