General Matrix Multiply (GEMM)

General Matrix Multiply (GEMM) is the dense multiply-accumulate kernel that computes $C = A \times B$ for matrices of arbitrary sizes, and whose throughput is the standard performance reference for dense linear algebra.

Why It Exists

Neural networks are, at their core, a long chain of matrix multiplications: attention projections, MLP weights, and even convolution (via im2col or implicit-GEMM). If the GEMM kernel is slow, the whole model is slow. Optimized GEMM kernels reach 80–90% of peak GPU throughput — the number FlashAttention-3 compares attention against — so GEMM is the roofline target that every other kernel is measured by.

How It Works

Each output element is a length-$K$ inner product:

\[C[m][n] = \sum_{k=0}^{K-1} A[m][k] \cdot B[k][n]\]

The full product is $M \times N \times K$ multiply-accumulate operations. Because that work dwarfs the data size, the practical problem is data movement, not FLOPs: operands must be fetched from global memory into on-chip storage and reused many times. GEMM implementations therefore break the computation into blocks and pick an accumulation strategy. The Spatial tutorial enumerates the main options: matrix tiling to block the loops, inner-product or outer-product accumulation, and systolic array dataflow. On hardware, tensor cores (NVIDIA), the Ascend Cube unit, and TPU systolic arrays execute GEMM as a single fused instruction with an FP32-accumulate epilogue.

Tradeoffs

GEMM is compute-bound only when tiles are large enough to amortize memory traffic; small matrices are memory-bound and need kernel fusion or batched dispatch.
Tiling must match the on-chip capacity (memory banking and SRAM size); too-large tiles spill, too-small tiles underutilize the compute units.
Low-precision GEMMs (FP8, FP4, INT4) multiply memory savings against accuracy; the NVFP4 format exists precisely to keep GEMM layout constraints tractable.

Common Confusions

GEMM vs. matmul: The same operation; GEMM emphasizes the general (non-square, non-identity) case with an optional bias/epilogue, as in cuBLAS GEMM.
GEMM vs. dot product: A GEMM is a matrix of inner products; a single dot product is one output element.
GEMM vs. convolution: Dense GEMM is a distinct kernel; convolution is often lowered to GEMM (implicit-GEMM) rather than being a GEMM itself.

Where It Appears

CUDA Programming Model: From Host to SM, Warp, and Memory — Uses vector addition and tile programming to show how device-memory traffic and on-chip reuse shape kernel cost.
CUDA Tile IR: The Design Philosophy of Tile Programming — Uses dynamic GEMM to show tile-block decomposition, tensor views, partition views, and compiler-owned mapping.
Spatial: General Matrix Multiply tutorial — Builds a blocked, outer-product GEMM in the Spatial DSL and discusses inner-product and systolic-array variants plus SRAM banking.
Spatial GEMM: Blocked Outer-Product Matrix Multiply — The knowledge-base insight page for the same tutorial: output tiling, MemFold/MemReduce, triple buffering, and banking.
FlashAttention-3 — Measures attention against optimized GEMM kernels that reach 80–90% of H100 peak, and overlaps GEMM with softmax.
MiniMax GQA W4A4 Quantization Path — W4A4 inference runs FP4 GEMM kernels (FlashInfer MXFP4, CUTLASS NVFP4) with dynamic activation quantization.
FlatQuant — Quantizes activations to INT4 inside a fused affine-transform kernel, then multiplies by pre-transformed INT4 weights with a CUTLASS GEMM.
Triton Ascend Operator Mechanisms — The Ascend Cube unit executes GEMM / QK·PV matrix multiply-accumulate.
NVFP4 — Defines GEMM layout constraints (TN-only) for 4-bit float weights and activations.
HiFloat4 (HiF4) — Designs a 64-length dot-product flow that keeps local scaling as shifts and delays shared floating-point scale work.
DeepGEMM MegaMoE: Fused Communication and Expert Compute — Uses two block-scaled expert GEMMs, SwiGLU, and SM100 tensor-memory staging inside a communication-overlapped MoE kernel.
Megatron-LM — Splits transformer GEMMs column- and row-wise across GPUs for tensor parallelism.
FlashAttention-2: Better Parallelism and Work Partitioning — FlashAttention-2 algorithm: reduced non-matmul overhead, sequence-parallel attention blocks, warp-level work partitioning, and.
FlashAttention-4: Blackwell Attention Kernel Co-Design — FlashAttention-4 algorithm and kernel-pipeline techniques for faster exact attention on NVIDIA Blackwell GPUs.
FlashAttention — The FlashAttention algorithm and kernel family across GPU generations: IO-aware exact attention, parallelism, Hopper asynchrony.
Kronecker Product — The block-structured matrix product A⊗B (matrix direct product), foundational to tensor factorization and the Kronecker.
DeepSeek V4 Attention: Code Reading Map — A navigable map of the DeepSeek V4 hybrid compressed attention implementation across vLLM (NVIDIA/AMD/XPU) and vllm-ascend.
Triton: Tiled GPU Kernel Language and Compiler — The original Triton language and compiler for expressing tiled neural network computations as portable, high-performance GPU.
Triton in Practice: How vLLM and vllm-ascend Use Triton — A codebase-driven tour of Triton kernel development in vLLM (NVIDIA GPU) and vllm-ascend (Ascend NPU), covering infrastructure.
TileLang Design and Code Learning Path — Uses a tiled GEMM to connect TileLang’s Python DSL to TIRX, target-specific lowering, and runtime adapters.
TileLang-Ascend: Ascend Backend and TileLang Integration — Uses Ascend GEMM as the concrete path from TileLang IR through Ascend C/PTO codegen and the CANN runtime.
CANNBot Skills: Triton Ascend Development Workflow — How CANNBot’s seven Triton-domain skills and the triton-op-generator plugin orchestrate end-to-end Triton Ascend kernel.
Triton Ascend: Ascend NPU Backend for Triton — A beginner-friendly tour of triton-ascend: how it bridges the Triton GPU kernel language to Huawei Ascend NPU hardware through a.
vLLM Kimi K3 Code Reading Map — Code-reading map for upstream vLLM’s real Kimi K3 implementation: request parsing, multimodal wrapper, KimiLinear text model.
DeepSeek-V4 Inference on Ascend: The DSA Serving Stack in vllm-ascend — How vllm-ascend runs DeepSeek-V4 end to end on Ascend NPUs: model override with mHC hyper-connections, hybrid c4/c128 compressor.
DeepSeek-V4 Lightning Indexer C8 Quantization: INT8/FP8 Indexer Cache in vllm-ascend — How vllm-ascend quantizes the DeepSeek-V4 Lightning Indexer to 8 bits (C8): INT8 keys with FP16 scales on 910B/A2/A3, FP8 e4m3fn.
Quantization — Post-training quantization methods and low-precision numeric formats for LLM inference.
Microscaling (MX) Formats: Block Floating Point for AI Hardware — Uses MX blocks as the narrow-precision operands inside matrix multiplies and describes the format’s accumulation and layout boundaries.
Gated Delta Networks: Improving Mamba2 with Delta Rule — Gated DeltaNet combines global state decay with key-targeted delta updates to improve fixed-state sequence memory while.
Kimi Linear: Expressive Efficient Attention Architecture — Kimi Linear is a hybrid linear attention architecture that for the first time outperforms full attention across short-context.
mHC: Manifold-Constrained Hyper-Connections — DeepSeek’s mHC projects Hyper-Connections’ residual mixing matrix onto the doubly stochastic manifold with Sinkhorn-Knopp.
Training Parallelism — Data, tensor, pipeline, and sequence parallelism techniques for large-model training.

Matrix Tiling — The blocking strategy that makes GEMM compute-bound.
Inner Product — Element-wise accumulation view of GEMM.
Outer Product — Rank-1 accumulation view of GEMM.
Systolic Array — A spatial dataflow that executes GEMM with maximal weight reuse.
Memory Banking — Conflict-free parallel access to the on-chip SRAM holding GEMM tiles.