Context Parallelism

Context Parallelism is a distributed inference strategy that shards a request’s sequence tokens and KV-cache entries across ranks while reconstructing exact attention through communication.

Why It Exists

Long-context prefill has both quadratic attention work and linearly growing KV-cache storage. Context parallelism adds hosts to the sequence dimension so local attention and cache capacity scale with the number of ranks.

How It Works

Each rank keeps the model’s TP-sharded weights and a slice of the request. A ring circulates either KV blocks or query blocks, depending on which message is cheaper for the current new-token length and cache-hit rate. Partial softmax outputs are merged with log-sum-exp statistics, so the result is exact dense attention rather than a sparse approximation.

Tradeoffs

Context parallelism is strongest for long prefill, where attention compute can hide ring communication. Decode has less compute to overlap, and query padding plus an output All-to-All can make time-to-incremental-token worse as the CP group grows. CP also replicates the model across hosts and therefore complements rather than replaces tensor parallelism.

Common Confusions

• Context parallelism vs. sequence parallelism: Context parallelism here targets inference, persistent KV state, and multi-turn decode; sequence parallelism is primarily a training strategy with a related sequence-sharding idea.
• Context parallelism vs. sparse attention: CP preserves every exact attention interaction; sparse attention reduces the number of interactions.

Where It Appears

• Context Parallelism for Scalable Million-Token Inference - Exact pass-KV and pass-Q ring attention for Llama3 405B, scaling to 1M-token prefill on 128 H100 GPUs.

• vLLM DCP and PCP: Decode and Prefill Context Parallelism - Separates prefill batch partitioning from decode KV sharding in the V1 serving runtime.

• vLLM DCP Attention: From Local LSE to Exact Global Output - Shows how DCP merges local outputs and LSE statistics to recover exact global attention.

• vLLM-Ascend Architecture: How the Ascend NPU Port Integrates with vLLM
• vLLM Kimi K3 Code Reading Map
• Kimi K3: Open 3T-Class Frontier Model

Related Terms

• Ring Attention - The blockwise communication pattern used to compute attention across shards.
• Sequence Parallelism - The training-oriented predecessor with a related sequence-dimension split.
• Tensor Parallelism - The complementary strategy that shards model weights.
• KV Cache - The request-specific state distributed by context parallelism.