Knowledge note

Matrix Tiling

Blocking a GEMM (or any kernel) into tiles that fit on-chip SRAM and registers so operands are loaded from global memory few times and reused many times.

Updated September 16, 2026Confidence: High

Matrix Tiling

Matrix Tiling (blocking) decomposes a large matrix computation such as GEMM into rectangular tiles that fit on-chip storage — SRAM, shared memory, or registers — so each operand is fetched from global memory a few times and reused many times.

Why It Exists

Without tiling, the inner loops of a GEMM re-read operands from slow global memory for every multiply, and the kernel is throttled by memory bandwidth instead of compute. The whole point of the memory hierarchy is reuse: if a tile of $A$ is loaded once into SRAM and multiplied against many tiles of $B$, the cost of loading it is amortized across thousands of multiply-accumulates. Every serious GEMM — cuBLAS, Triton, FlashAttention, Spatial’s blocked GEMM — is a tiling strategy.

How It Works

The output is partitioned into $M/bm \times N/bn$ tiles of C, and the shared dimension into $P/bp$ chunks. For each output tile:

  1. Prefetch the C tile into SRAM.
  2. For each $k$-chunk, load a $bm \times bp$ tile of A and a $bp \times bn$ tile of B in parallel, then run the tile-level multiply-accumulate.
  3. Store the accumulated C tile back to global memory.

The Spatial tutorial expresses this with Foreach(M by bm, N by bn) over output tiles, a MemFold over $P$ chunks that accumulates on top of the prefetched C tile, and a MemReduce inside that accumulates the per-$k$ partial tiles. A triple-buffered memory overlaps the load, compute, and store stages of the outermost pipeline so the compute unit never waits on memory. Hierarchical tiling (as in Triton) layers this: shared-memory-sized micro-tiles and register-sized nano-tiles.

Tradeoffs

  • Larger tiles increase arithmetic intensity but must fit on-chip; exceeding SRAM/register capacity causes spills and serialization.
  • Tile shape interacts with layout (row-major vs. column-major) and with memory banking: the banking scheme must allow every parallel access into a tile to hit a distinct bank.
  • The loop order (which tiling dimension is outermost) changes DRAM traffic; Spatial scans output tiles horizontally to minimize transactions.

Common Confusions

  • Tiling vs. blocking: The same idea; “blocking” is the classic high-performance-computing term, “tiling” the accelerator/GPU term.
  • Tiling vs. memory banking: Tiling decides which data is on-chip and how loops are ordered; banking decides how parallel accesses to that on-chip data avoid collisions.
  • Tiling vs. kernel fusion: Fusion avoids writing intermediate tiles to global memory between kernels; tiling keeps individual kernels’ operands on-chip.

Where It Appears