Knowledge note
Continuous Batching
An LLM-serving scheduling pattern that rebuilds the active batch at each model iteration so finished requests can leave and newly ready work can enter.
Continuous Batching
Continuous Batching is an LLM-serving scheduling pattern that chooses the active work again at every model iteration, allowing completed requests to leave and newly ready requests to enter without a whole-batch barrier.
Why It Exists
Autoregressive requests generate different numbers of tokens. In a static batch, short requests leave idle rows until the longest request finishes; continuous batching turns each decode boundary into another scheduling opportunity.
How It Works
The scheduler tracks live request state across iterations. Before each model forward, it selects runnable requests and assigns per-request token work under compute and memory limits. After the forward, it advances each request independently, removes finished work, and repeats.
Modern engines may mix one-token decodes, prompt chunks, and speculative tokens in one iteration. The mechanism is therefore more precise than the simplified picture of replacing only a finished decode row.
Tradeoffs
Continuous batching adds scheduler and batch-reconciliation overhead. Under high load, prompt work can interfere with decode latency, and KV-cache pressure can cause preemption or recomputation. It provides little benefit when traffic is sparse or request lengths are uniform.
Common Confusions
- Continuous vs. request-level dynamic batching: Request-level batching chooses a static group before execution; continuous batching can change the group between generation iterations.
- Continuous batching vs. PagedAttention: Scheduling decides when work runs; PagedAttention decides how its KV state is allocated and accessed.
- Continuous batching vs. chunked prefill: Chunking splits one prompt over iterations; continuous batching is the broader policy that chooses all work in each iteration.
Where It Appears
- vLLM: PagedAttention Serving Framework — Relates Orca-style iteration-level scheduling to vLLM’s paged KV-memory management.
- vLLM Continuous Batching — Traces the current V1 scheduler, KV allocation, worker-batch update, finish, and preemption paths.
- vLLM Block Table Management — Connects iteration-level scheduling to paged KV-block allocation.
- vLLM-Ascend Architecture — Shows how the Ascend backend reuses vLLM’s continuous-batching scheduler.
- Sarathi: Chunked Prefills for Efficient LLM Inference — Shapes continuous work into chunked prefill and decode-maximal batches.
- vLLM-Ascend Prefill and Decode Scheduling: Qwen3.5 GQA — Shows how the upstream continuous-batching plan becomes Ascend FIA/GDN execution metadata.
- vLLM — vLLM serving framework pages: PagedAttention paper, request-flow code learning path, continuous batching, and Kimi K3 code.
- vLLM Architecture and Code Organization Overview — A top-down code-reading map of the vLLM repository at commit a0c092ee72c0: how the V1 serving engine, model executor, config.
- vLLM DCP and PCP: Decode and Prefill Context Parallelism — PCP rewrites the continuous batch into rank-local prefill chunks and replicated decodes.
Related Terms
- KV Cache — Request state whose capacity often limits continuous-batch concurrency.