Pipeline Parallelism

Pipeline Parallelism is a model-parallel strategy that assigns different layer ranges to stages and streams microbatches through those stages.

Why It Exists

A model can be too large for one device even after tensor sharding, while independent microbatches provide enough work to keep several layer stages busy. Pipeline parallelism addresses model capacity and aggregate throughput.

How It Works

Stage $i$ runs its assigned layers, sends activations to stage $i+1$, and receives the next microbatch from stage $i-1$. Schedules overlap microbatches, but warmup and drain periods leave pipeline bubbles.

Tradeoffs

Pipeline parallelism does not split the tokens of one request across stages, so it is not a direct solution to long-context prefill latency. It composes with context parallelism: PP can divide layers while CP divides sequence state.

Common Confusions

• Pipeline vs. tensor parallelism: Pipeline parallelism splits by layer depth; tensor parallelism splits each layer’s matrix operations.
• Pipeline vs. context parallelism: Pipeline parallelism mainly improves throughput and model fit; context parallelism targets sequence-length latency and KV capacity.

Where It Appears

• Context Parallelism for Scalable Million-Token Inference - Describes why pipeline parallelism alone does not remove long-prompt latency.

• vLLM DCP and PCP: Decode and Prefill Context Parallelism — PCP currently rejects PP greater than one in its MRV2 validation.

• Sarathi: Chunked Prefills for Efficient LLM Inference
• mHC: Manifold-Constrained Hyper-Connections
• GPipe: Micro-Batch Pipeline Parallelism
• Training Parallelism
• Megatron-LM: GPU-Cluster Training Parallelism
• Sequence Parallelism: Splitting Sequences Across GPUs

Related Terms

• Tensor Parallelism - Splits layer computation across ranks.
• Context Parallelism - Splits sequence state across ranks.
• Microbatch - The unit streamed through pipeline stages.