NVFP4

NVFP4 is NVIDIA’s four-bit floating-point format for Blackwell-era tensor computation, combining E2M1 payload values with fractional FP8 E4M3 micro-block scales and a tensor-level scale in the public recipe.

Why It Exists

Plain FP4 has too few representable values for model weights and activations. NVFP4 uses a scale for each 16-value micro-block so the E2M1 payload can fit the local magnitude more closely than a single power-of-two scale.

How It Works

The public NVIDIA recipe reconstructs values from a four-bit E2M1 payload, an E4M3 scale for each 16-element block, and an FP32 scale for the tensor. Training adds recipe-level mechanisms such as stochastic rounding, 2D weight scaling, and Random Hadamard Transform support; inference and training also depend on the supported Blackwell GEMM layout.

Tradeoffs

Fractional scales and small blocks improve local fitting but require more scale handling and can need global scaling to keep E4M3 in range. The format is not a drop-in software type: hardware support, layout, conversion policy, and workload distribution determine the actual benefit.

Common Confusions

NVFP4 vs. HiF4: Both cost about 4.5 bits/value in the compared designs, but NVFP4 uses 16-value E2M1 blocks with fractional scales, while HiF4 uses 64 S1P2 values plus shared E1 micro-exponents.
NVFP4 vs. generic FP4: NVFP4 is a hierarchical scaled format and deployment recipe, not just an unscaled four-bit E2M1 scalar.

Where It Appears

NVFP4: Blackwell 4-Bit Floating Point — Public Transformer Engine and NVIDIA blog recipe, training mechanisms, layout limits, and deployment context.
HiFloat4 (HiF4): 4-Bit Block Floating Point for LLM Inference — Compares NVFP4 against a 64-value shared-metadata alternative for error, hardware, and LLM accuracy.
Microscaling (MX) Formats: Block Floating Point for AI Hardware — Places NVFP4 alongside OCP MX and related block-floating designs.