FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

The Core Problem and Solution

Attention—the mechanism that lets language models weigh which parts of input text matter most—has quadratic complexity: doubling your context length quadruples the computation. For long-context models processing 128K tokens or more, the prefilling phase (where the model processes the full prompt before generating output) becomes a severe bottleneck. FlashPrefill V2 solves this by making attention sparse: it skips most token pairs and only computes attention between tokens that actually matter, cutting prefill time dramatically.

The trick is deciding which token pairs to attend to without being too conservative. The original FlashPrefill used "instantaneous pattern discovery"—essentially finding dense blocks in the attention matrix on the fly—but remained a prototype unsuitable for real-world deployment. V2 productionizes this idea.

How It Works

FlashPrefill V2 identifies sparse attention patterns using a max-based dynamic threshold: it computes a small sample of attention scores, finds the maximum, and attends only to tokens that exceed a threshold relative to that max. This discovers locality and long-range structure automatically without hand-crafted sparsity patterns. The authors then add a mean correction term that calibrates for the bias introduced by pruning, keeping accuracy loss minimal even when attention becomes very sparse.

On the implementation side, the authors redesigned the sparse attention operator to match modern GPU efficiency standards. Key optimizations include:

• PackGQA memory access: Groups queries and key-value pairs efficiently to reduce memory traffic.
• Warp specialization: Different GPU thread groups handle different computational tasks optimally.
• Pingpong pipelining: Overlaps computation and memory transfers to hide latency.

These align with FlashAttention-3/4, the current state-of-the-art dense attention kernel. Critically, FlashPrefill V2 also supports FP8 quantization (8-bit floating point), a practical requirement for cost-effective inference at scale. The system integrates with paged KV cache and continuous batching—standard production features that allow multiple requests to share GPU memory and be processed together—making it compatible with inference frameworks like SGLang.

Why It Matters

The speedups are substantial. On NVIDIA H20 GPUs (among the most deployed inference accelerators), FlashPrefill V2 achieves 47.26× faster prefill than FlashAttention-2 in FP8 and 27.19× faster in BF16 (a lower-precision format) at 128K context length. Even against a state-of-the-art dense baseline aligned with FlashAttention-3/4, it still delivers 30.49× speedup in FP8.

This gap matters because prefilling is often the latency bottleneck in long-context serving. For applications like document retrieval, code analysis, or multi-turn dialogue with long histories, cutting prefill time by 30–47× directly improves user-facing latency and GPU throughput.

The authors move FlashPrefill from research prototype to a production-ready system: it handles quantization, integrates with standard batching schemes, and runs on widely used hardware. This lowers the barrier for practitioners to deploy long-context models efficiently at scale.