FP8

FP8 is an 8-bit floating-point representation used to reduce the storage and movement cost of neural-network tensors.

Why It Exists

Large-model inference is constrained by both device memory and bandwidth. Replacing 16-bit values with scaled 8-bit values can make a model fit and can leave more bandwidth for attention communication.

How It Works

An FP8 deployment chooses an encoding such as E4M3 or E5M2 and applies scale factors so the representable range and precision match the tensor. The context-parallel paper uses row-wise FP8 weights for Llama3 405B’s feed-forward layers while evaluating the parallel attention system.

Tradeoffs

FP8 is a numerical optimization, not a context-parallel algorithm. Scale selection, outliers, accumulation precision, and hardware support determine whether it preserves quality and delivers a speedup.

Common Confusions

• FP8 vs. INT8: FP8 preserves a floating-point exponent and fraction; INT8 uses an integer range and a different scaling model.
• FP8 vs. exact attention: The paper’s attention communication algorithm is exact relative to the evaluated FP8 model; FP8 does not make the attention sparse.

Where It Appears

• Context Parallelism for Scalable Million-Token Inference - Uses row-wise FP8 weights in the Llama3 405B benchmark configuration.

• DeepSeek-V2 Multi-Head Latent Attention
• DeepSeek-V3.2: Sparse Attention, Scaled RL, and Thinking in Tool-Use
• FlashAttention-2: Better Parallelism and Work Partitioning
• FlashAttention-3: Hopper Asynchrony and FP8 Attention
• FlashAttention
• DeepSeek V4 Attention: Code Reading Map
• Triton in Practice: How vLLM and vllm-ascend Use Triton
• vLLM-Ascend Architecture: How the Ascend NPU Port Integrates with vLLM
• DeepSeek-V4 Inference on Ascend: The DSA Serving Stack in vllm-ascend
• DeepSeek-V4 Lightning Indexer C8 Quantization: INT8/FP8 Indexer Cache in vllm-ascend
• vLLM Ascend
• Qwen3.5 / Qwen3.6 Inference Path on vLLM Ascend
• MiniMax GQA W4A4 Quantization Path: GPU (vLLM) and NPU (vllm-ascend)
• vLLM Block Table Management: From PagedAttention to the V1 KV Cache Stack
• vLLM Kimi K3 Code Reading Map
• FlatQuant: Fast Learnable Affine Quantization
• Quantization
• Microscaling (MX) Formats: Block Floating Point for AI Hardware
• NVFP4: Blackwell 4-Bit Floating Point
• DeepGEMM MegaMoE: Fused Communication and Expert Compute - Uses FP8 activations, FP8 scale factors, and packed FP4 expert weights in an SM100 block-scaled MoE path.
• NVIDIA Ada Lovelace Professional GPU Architecture - Describes FP8 as a fourth-generation Tensor Core capability in a professional GPU.
• DeepSeek-V4: Million-Token Context via Hybrid Compressed Attention
• MiniMax Sparse Attention (MSA)

Related Terms

• Microscaling - A block-level scaling family for low-precision formats.
• NVFP4 - A four-bit floating-point deployment format.