Knowledge note
Training Parallelism
Data, tensor, pipeline, and sequence parallelism techniques for large-model training.
Training Parallelism
- Megatron-LM: GPU-Cluster Training Parallelism — Covers both Megatron-LM papers: intra-layer tensor parallelism with f/g conjugate operators and column-parallel GEMM splitting (2019), and the PTD-P recipe for trillion-parameter GPT models (2021).
- GPipe: Micro-Batch Pipeline Parallelism — Synchronous micro-batch pipeline parallelism with activation recomputation: splits mini-batches into micro-batches, pipelines them through partitioned layers, and applies synchronous gradient updates for near-linear speedup.
- Sequence Parallelism: Splitting Sequences Across GPUs — Sequence parallelism: distributes input sequence chunks across GPUs with Ring Self-Attention (RSA), introducing the fourth parallelism dimension alongside data, pipeline, and tensor parallelism.