Knowledge note

Chunked KV-Cache Compression

Stores fewer key/value entries by summarizing consecutive token windows at a fixed stride, trading long-context cache cost for phase-dependent retention.

Updated October 10, 2026Confidence: High

Chunked KV-Cache Compression

Chunked KV-Cache Compression maps consecutive token windows into fewer cached key/value entries, reducing long-context state while making each token’s representation depend on its position in the compression cycle.

Why It Exists

An ordinary KV cache retains keys and values for every token, so its memory grows with context length. A compressor can combine multiple neighboring positions into a smaller set of entries before a future query is known, reducing cache size and the number of positions later attention must read.

How It Works

A model groups source tokens into windows of width $W$ and emits compressed entries at stride $S$. Learned payload maps and gates, or a simpler aggregation such as uniform averaging, form the compressed keys and values. The source phase $\phi = t \bmod S$ identifies where a token at position $t$ falls relative to the stride; shifting the prefix can change its summary even when the token and its relationship to nearby content stay the same.

With non-overlapping windows, $W=S$. Overlapping designs can use $S<W$, so a token participates in multiple windows while its phase still follows the stride. Compression is distinct from sparse attention: compression changes how many key/value entries are stored, while sparsity selects which stored entries a query reads.

Tradeoffs

Fewer entries can reduce cache and attention cost, but summaries are formed before later query intent is known and may not preserve every source detail equally. Periodic Weak Spots measures phase-dependent retrieval gaps in chunk-compressed models; average accuracy alone may conceal a weak residue.

Common Confusions

  • Chunked KV-cache compression vs. cache quantization: Compression combines source positions into fewer entries; quantization stores each value with fewer bits.
  • Chunked KV-cache compression vs. sparse attention: Compression changes the stored representation; sparse attention selects a subset of stored positions for a query.
  • Window size vs. stride: $W$ is how many tokens a summary covers, while $S$ is how often the compressor advances. They are equal only in non-overlapping designs.

Where It Appears