- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 5
A conversation between
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
§02
Snippets
-
SeKV organizes context into entropy-guided semantic spans with lightweight GPU summaries and CPU-stored low-rank bases, enabling on-demand token reconstruction without full cache materialization.
Achieves 53% GPU memory reduction while recovering token-level detail on-the-fly, bridging compression efficiency and accuracy gaps.
-
A trained zoom-in mechanism selectively expands query-relevant spans during decoding, recovering precise information from compressed representations in real time.
Allows adaptive, lossless reconstruction mid-generation—compressed spans can yield token-level detail when a query suddenly needs it.
-
SeKV keeps the base LLM frozen while adding fewer than 0.05% trainable parameters in the zoom-in mechanism.
Enables drop-in deployment on existing models without expensive retraining or model modification.
-
GPU stores entropy-guided semantic span summaries for coarse routing; CPU stores low-rank SVD bases allowing selective reconstruction of individual tokens as queries arrive.
Couples efficiency (coarse GPU routing) with precision (fine-grained CPU recovery), adapting to emergent information needs during generation.
-
SeKV improves 5.9% over the strongest semantic compression baseline across four benchmarks while maintaining lossless information recovery.
Demonstrates that hierarchical storage and adaptive zoom-in outperform fixed-compression strategies in both typical and challenging scenarios.
§03
Synthesis
The Core Problem and Solution
Modern large language models face a memory crisis when processing long documents: storing all the key-value pairs (KV cache) needed for attention grows linearly with context length, quickly exhausting GPU memory. Existing fixes fall into two traps. Token eviction permanently deletes information, degrading quality. Semantic grouping—compressing similar tokens together at the start—locks in that decision and can't recover fine details later when they become important. SeKV solves this by keeping information intact but storing it smartly: summaries stay on GPU for fast routing, full token details live on CPU, and a learned mechanism decides when to "zoom in" and fetch the details.
How It Works
SeKV organizes the KV cache into semantic spans—clusters of similar tokens identified using entropy signals. Each span gets two representations:
- Lightweight GPU summary: A single vector per span that acts as a coarse fingerprint, enabling quick routing decisions without loading everything.
- Low-rank CPU storage: A truncated singular value decomposition (SVD) basis that compactly captures all token-level information in the span. Think of it as a compressed blueprint that can reconstruct individual tokens on demand.
During decoding, when the model generates a new token and needs to attend to the context, a trained "zoom-in" mechanism checks which spans might be relevant using the GPU summaries. If a span looks important, the model fetches its SVD basis from CPU and reconstructs the specific tokens it needs. This selective expansion happens only for spans that matter for the current query, avoiding full GPU materialization.
The key insight: the model doesn't throw away tokens (unlike eviction methods) and doesn't freeze decisions at prefill time (unlike fixed grouping). It operates at variable resolution—coarse when scanning, fine when needed.
Training and Scale
The method adds a lightweight trainable component for the zoom-in decision mechanism, contributing fewer than 0.05% additional parameters while keeping the base LLM fully frozen. This makes it practical to deploy on existing models without expensive retraining.
Empirical Results
Tested on four benchmarks with 128K token contexts, SeKV improves over the strongest semantic compression baseline by 5.9% on average—a meaningful gain when a baseline already uses compression. Versus storing the full KV cache on GPU, it cuts GPU memory by 53.3%, a critical reduction for scaling to real long-document tasks. The resolution-adaptive design means it doesn't sacrifice quality to achieve that compression; it adapts granularity based on what the model actually needs.
Why It Matters
As LLMs tackle longer contexts (legal documents, codebases, conversations), GPU memory becomes the binding constraint on inference speed and cost. SeKV sidesteps the false choice between fidelity and efficiency by leveraging the memory hierarchy (GPU + CPU) and an adaptive mechanism to retrieve information only when queries demand it. The frozen base model and minimal parameters make adoption straightforward for practitioners.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.