Tensor Parallelism

Tensor Parallelism is a model-parallel strategy that splits a layer’s weight matrices or hidden/head dimensions across accelerator ranks.

Why It Exists

Large models may not fit on one accelerator, and even when they do, splitting matrix multiplications can reduce per-device compute time. The price is communication after partial results must be combined.

How It Works

Column- and row-parallel linear layers divide the matrix work across ranks and use collectives such as all-reduce to make the next layer see the correct result. In the context-parallel paper, TP8 stays within each host while CP adds hosts along the sequence dimension.

Tradeoffs

Tensor parallelism shards model weights, which CP does not, but its frequent collectives are costly across slower inter-host links. The useful TP degree is also constrained by hidden dimensions, attention heads, and the interconnect topology.

Common Confusions

• Tensor vs. context parallelism: Tensor parallelism splits model computation; context parallelism splits request tokens and KV state.
• Tensor vs. pipeline parallelism: Tensor parallelism splits each layer across ranks; pipeline parallelism assigns different layers to stages.

Where It Appears

• Context Parallelism for Scalable Million-Token Inference - Uses TP8 within each host and CP across hosts to balance model fit with long-context latency.

• vLLM DCP and PCP: Decode and Prefill Context Parallelism — DCP reuses TP ranks when PCP is disabled and may span the full TP x PCP group.

• vLLM-Ascend Architecture: How the Ascend NPU Port Integrates with vLLM
• vLLM: PagedAttention Serving Framework
• NVFP4: Blackwell 4-Bit Floating Point
• Training Parallelism
• Megatron-LM: GPU-Cluster Training Parallelism
• Sequence Parallelism: Splitting Sequences Across GPUs

Related Terms

• Context Parallelism - Splits sequence state rather than model weights.
• Pipeline Parallelism - Splits layers into stages.
• All-Reduce - Common collective for combining tensor-parallel partial results.