KV Cache

KV Cache is the per-layer store of attention keys and values produced by earlier tokens, reused during autoregressive decoding so the model does not recompute the entire prefix.

Why It Exists

Without caching, generating token $i$ repeats key and value projections for tokens $1$ through $i-1$. Caching makes those projections persistent and reduces the repeated work, which is essential for practical Transformer decoding.

How It Works

Each new token appends one key vector and one value vector per relevant layer and KV head. Its query attends over the cached keys and combines the corresponding values. Projection work per step becomes constant, but cache memory and the attention read still grow linearly with context length.

Tradeoffs

Long contexts can make the cache the dominant memory consumer and constrain batch size. MQA, GQA, MLA, quantization, paging, sparse attention, and linear attention reduce different parts of this cost; only recurrent-state approaches make stored state independent of sequence length.

Common Confusions

• KV cache vs. model weights: Weights are fixed learned parameters; the KV cache is request-specific runtime state.
• KV cache vs. linear-attention state: A KV cache preserves token-level keys and values; a linear-attention state merges them into a fixed-size summary.

Where It Appears

• Transformers Are RNNs: Linear Attention — Contrasts explicit growing key/value history with a fixed-size recurrent summary.
• Kimi Linear — Uses recurrent KDA states in most layers and full KV caches in periodic MLA layers.
• Kimi K3 — Adds external KV-cache retention for million-token partial rollouts and aligns KDA state lifecycles with MLA cache blocks.
• vLLM Continuous Batching — Shows how paged KV capacity gates per-iteration admission, completion, and preemption.
• vLLM Block Table Management — Details how paged KV blocks, reference counts, and prefix caching implement the cache’s paged storage.
• FlatQuant — Applies per-head learnable affine transforms to keys and values before low-bit KV-cache quantization (down to 2 bits).
• vLLM-Ascend Architecture — Shows how Ascend attention backends allocate and consume vLLM-compatible KV caches.
• DeepSeek-V2 Multi-Head Latent Attention — Explains DeepSeek-V2’s Multi-head Latent Attention: low-rank joint key/value compression, decoupled RoPE, MoE-scale inference.
• Grouped-Query Attention in Llama 2 — Explains why Llama 2 uses grouped-query attention for its 34B and 70B models: it cuts KV-cache pressure like multi-query.
• Multi-Query Attention: One Write-Head is All You Need — Replaces per-head key/value projections with a single shared K/V pair across all attention heads, eliminating the heads.
• Recurrent Neural Networks: From RNN to LSTM — A beginner-oriented explanation of recurrent hidden state, shared weights across sequence steps, long-term dependency failures.
• 线性Attention的探索：Attention必须有个Softmax吗？ — Su Jianlin’s influential blog survey on why softmax is the bottleneck of standard attention, how removing it enables O(n) linear.
• AISBench Benchmark vs. EvalScope Perf — Competitive analysis of AISBench Benchmark and EvalScope Perf across load generation, workloads, metrics, extensibility.
• DeepSeek — DeepSeek model implementation readings that span vLLM and vllm-ascend codebases.
• DeepSeek V4 Attention: Code Reading Map — A navigable map of the DeepSeek V4 hybrid compressed attention implementation across vLLM (NVIDIA/AMD/XPU) and vllm-ascend.
• KV Cache Optimization Strategies for Scalable and Efficient LLM Inference — Surveys the five main ways to reduce KV-cache pressure and maps them to deployment scenarios.
• KVSwap: Disk-Aware KV Cache Offloading for Long-Context On-Device Inference — Keeps the full cache on disk and uses a compact predictor, grouped reads, and reuse to support long-context decoding on constrained devices.
• Sarathi: Chunked Prefills for Efficient LLM Inference — Sarathi improves LLM serving by splitting prefills into compute-sized chunks and piggybacking decode tokens on them to raise.
• SGLang: Structured Language Model Programs — SGLang framework architecture, programming model, runtime optimizations, and evaluation results for efficient structured LLM.
• Triton in Practice: How vLLM and vllm-ascend Use Triton — A codebase-driven tour of Triton kernel development in vLLM (NVIDIA GPU) and vllm-ascend (Ascend NPU), covering infrastructure.
• Triton Ascend Operator Mechanisms: Vector, Cube, and CV Fusion — A practical learning path for Triton Ascend Vector, Cube, and CV fusion operators, grounded in AI Core compute units, on-chip.
• vLLM — vLLM serving framework pages: PagedAttention paper, continuous batching, and Kimi K3 code.
• vLLM: PagedAttention Serving Framework — vLLM framework design, PagedAttention memory management, scheduling, decoding support, and serving performance results.
• vLLM Kimi K3 Code Reading Map — Code-reading map for upstream vLLM’s real Kimi K3 implementation: request parsing, multimodal wrapper, KimiLinear text model.
• vLLM Architecture and Code Organization Overview — A top-down code-reading map of the vLLM repository at commit a0c092ee72c0: how the V1 serving engine, model executor, config.
• DeepSeek-V4 Inference on Ascend: The DSA Serving Stack in vllm-ascend — How vllm-ascend runs DeepSeek-V4 end to end on Ascend NPUs: model override with mHC hyper-connections, hybrid c4/c128 compressor.
• DeepSeek-V4 Lightning Indexer C8 Quantization: INT8/FP8 Indexer Cache in vllm-ascend — How vllm-ascend quantizes the DeepSeek-V4 Lightning Indexer to 8 bits (C8): INT8 keys with FP16 scales on 910B/A2/A3, FP8 e4m3fn.
• vLLM Ascend — vLLM’s Ascend NPU port: code-reading notes and MoE forward implementation insights.
• Qwen3.5 / Qwen3.6 Inference Path on vLLM Ascend — A code-reading tour of the shared qwen3_5-family inference path: Qwen3.5-27B / Qwen3.6-27B (dense hybrid Mamba-Transformer.
• DeepSeek-V4: Million-Token Context via Hybrid Compressed Attention — DeepSeek-V4 introduces hybrid Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), Manifold-Constrained.
• DeepSeek-V4.1-Flash — Uses CED, cross-layer CSA2 reuse, FP4 main KV, and bounded SWA replay to reduce global and persistent cache footprints.
• DeepSeek — DeepSeek model training papers: V4 hybrid compressed attention and V3.2 sparse attention with scaled RL.
• YOCO: You Only Cache Once — Separates a bounded-memory self-decoder from cross-decoder layers that all reuse one global KV cache.
• Megatron-LM: GPU-Cluster Training Parallelism — Explains both Megatron-LM papers: intra-layer tensor model parallelism with f/g conjugate operators (8.3B, V100), and the PTD-P.
• Context Parallelism for Scalable Million-Token Inference — Shards persistent KV state across context-parallel ranks and chooses whether to circulate KV or queries.
• vLLM DCP and PCP: Decode and Prefill Context Parallelism — DCP shards persistent decode KV state; PCP gathers prefill cache inputs without duplicating decode writes.

• vLLM Prefill/Decode Disaggregated Deployment Path — Transfers request-specific paged KV blocks from an independently scaled prefill pool into decode-owned storage.

• vLLM DCP Attention: From Local LSE to Exact Global Output

Related Terms

• Linear Attention — Replaces explicit token history with accumulated key–value statistics.
• Kimi Delta Attention — A Kimi-family fixed-state attention mechanism used to reduce cache pressure.
• Continuous Batching — Uses iteration-level admission to keep available KV capacity productive.
• PagedAttention — The algorithm that stores KV cache in fixed-size non-contiguous blocks.
• Block Table — The logical-to-physical mapping used to address paged KV cache.