Knowledge note
Chunked Prefill
Splitting a long LLM prompt into causal chunks processed over multiple iterations so prompt work can share batches with decode work.
Chunked Prefill
Chunked prefill splits a long LLM prompt into smaller causal chunks and processes those chunks across multiple model iterations instead of executing the entire prompt in one prefill pass.
Why It Exists
A full prefill can saturate accelerator compute, while decode work is often memory-bound at practical batch sizes. Chunking creates several prompt-work opportunities that can be mixed with decode tokens, improving utilization and making per-iteration work more predictable.
How It Works
Each chunk writes KV-cache entries for its prompt tokens. Later chunks attend to all preceding prompt keys and values, but not future tokens. With a correct causal attention mask, the final KV cache and model outputs are equivalent to processing the full prompt at once.
The chunk size is a tradeoff: smaller chunks provide more opportunities to interleave decodes but reduce matrix-multiplication efficiency and reread earlier KV entries more often.
Common Confusions
- Chunked prefill vs. continuous batching: Chunking splits one prompt; continuous batching chooses the complete set of work for each model iteration.
- Chunked prefill vs. decode: A chunk still processes prompt tokens in parallel; decode advances one generated token per request.
- Chunked prefill vs. paged KV memory: Chunking controls execution order; paged KV memory controls where cached keys and values are stored.
Where It Appears
- Sarathi: Chunked Prefills for Efficient LLM Inference - combines chunked prefills with decode-maximal batching and pipeline scheduling.
- vLLM Continuous Batching - implements prompt chunks under a token-budget scheduler in a modern serving engine.
- vLLM-Ascend Prefill and Decode Scheduling: Qwen3.5 GQA - follows the prompt-chunk cursor from vLLM scheduling into Ascend mixed FIA execution.
- vLLM — vLLM serving framework pages: PagedAttention paper, continuous batching, and Kimi K3 code.
- vLLM Architecture and Code Organization Overview — A top-down code-reading map of the vLLM repository at commit a0c092ee72c0: how the V1 serving engine, model executor, config.
- vLLM Prefill/Decode Disaggregated Deployment Path — Contrasts prompt chunking inside one mixed scheduler with isolating prefill and decode in separate engine pools.