Knowledge note
Grouped-Query Attention
An attention layout that shares each key/value head across a group of query heads to reduce KV-cache and communication cost.
Grouped-Query Attention
Grouped-Query Attention is an attention layout in which multiple query heads share a smaller set of key/value heads.
Why It Exists
Autoregressive inference stores and reads K/V for every layer and token. Reducing the number of KV heads cuts that memory traffic while retaining more query-head diversity than single-KV-head multi-query attention.
How It Works
If a model has $N_H$ query heads and $N_{KV}$ KV heads, each KV head serves approximately $N_H/N_{KV}$ query heads. In the paper’s Llama3 405B setup, 128 query heads share 8 KV heads, so a CP ring can communicate a much smaller KV payload than a query-shaped payload.
Tradeoffs
Fewer KV heads reduce cache and communication cost, but the model must be trained or adapted for the grouping. The ratio also changes context-parallel pass selection because it sets the relative size of Q versus KV messages.
Common Confusions
- GQA vs. MQA: GQA uses several KV heads; MQA uses one shared KV head for all query heads.
- GQA vs. context parallelism: GQA reduces the payload per token; context parallelism distributes tokens and communicates those payloads across ranks.
Where It Appears
- Context Parallelism for Scalable Million-Token Inference - Uses Llama3’s 128 query heads and 8 KV heads to make pass-KV communication efficient.
-
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints - Introduces grouped-query adaptation from multi-head checkpoints.
- Grouped-Query Attention in Llama 2
- Multi-Query Attention: One Write-Head is All You Need
- Qwen3.5 / Qwen3.6 Inference Path on vLLM Ascend
- MiniMax GQA W4A4 Quantization Path: GPU (vLLM) and NPU (vllm-ascend)
Related Terms
- KV Cache - Stores the grouped key/value states during decoding.
- Context Parallelism - Distributes the grouped KV state across ranks.