OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

The Core Problem: KV Cache is Strangling LLM Inference

When an LLM generates text token-by-token (the "decode" phase), it must store and access the key-value (KV) cache—a compressed representation of all previously generated tokens. For long contexts or extended reasoning, this cache explodes in size, consuming most of the expensive high-bandwidth memory (HBM) on GPUs. This memory bottleneck kills throughput: you can't fit many requests on a single GPU, even though the compute itself isn't saturated.

The Insight: Sparse Attention + Lookahead Prediction

The authors' key observation is that during decoding, not all past tokens matter equally for computing the next token's attention. In practice, only a small subset of tokens contribute significantly to the prediction.

OasisKV exploits this by:

1. Keeping only hot KV entries in HBM. Instead of storing the entire KV cache on the GPU, OasisKV maintains only the KV pairs for the most relevant tokens in fast HBM.

2. Predicting which tokens you'll need next. The system uses "lookahead tokens"—tokens drafted speculatively by existing speculative decoding methods—to predict which past tokens will be important in the next decode step. Because these drafts look slightly ahead, the system has time to stage the right KV data before it's needed.

3. Background prefetching pipeline. While the GPU computes the current token's attention over the hot KV entries, a background pipeline identifies which KV blocks will be needed next and pulls them from cheaper, higher-capacity memory (host RAM or remote memory) into HBM. This overlaps data movement with compute, hiding latency.

Why It Works and What It Delivers

The lookahead prediction is accurate enough that even with a tight 2,048-token KV budget, accuracy stays within 0.7 points of full dense attention. This sparsity translates directly to throughput:

• Single-GPU reasoning: 1.69× speedup over dense vLLM with only 0.1 points accuracy loss.
• Multi-GPU long-context serving: Up to 2.1× speedup.
• Disaggregated prefill-decode: ~2× throughput while using 6.5–9.7× less KV cache transfer per request and 2.2–2.6× less host memory on decode nodes.

The system is implemented on top of vLLM, an industry-standard inference framework, making it practical to deploy.

Why This Matters

As reasoning workloads (like chain-of-thought LLM inference) and long-context retrieval become standard, memory—not compute—is the real bottleneck. OasisKV shows that you can reclaim HBM capacity and boost system throughput by treating KV cache as a tiered resource: hot data in HBM, cold data in slower memory, with intelligent prefetching to mask the latency of pulling data back. This is a direct path to lower serving costs and higher request throughput for production LLM systems.