Ring Attention

Ring Attention is an exact attention schedule that circulates query or key/value blocks around a logical device ring and merges the resulting partial softmax computations.

Why It Exists

Long sequences create attention and memory blocks that are too large for one device, while a full all-gather can put communication directly on the critical path. A ring exposes one manageable block at a time and overlaps its transfer with attention work.

How It Works

In pass-KV, every rank keeps its query block and sends its KV block to a neighbor until all ranks have been visited. In pass-Q, KV stays resident and query blocks travel instead; partial outputs return to their source rank through an output redistribution. Each partial attention result carries a log-sum-exp statistic, allowing an exact stable merge across blocks.

Tradeoffs

Ring attention needs repeated point-to-point traffic and becomes communication-bound when local attention work is too small. Pass-Q saves bandwidth for cache-heavy decode but adds an All-to-All-style output exchange; pass-KV is often better for full prefill with many new queries.

Common Confusions

• Ring attention vs. sparse attention: Ring attention changes the schedule, not which tokens attend to which; it remains exact.
• Pass-KV vs. pass-Q: They are two traffic directions in the same exact ring family, selected for different query/KV size ratios.

Where It Appears

• Context Parallelism for Scalable Million-Token Inference - Adapts pass-KV and pass-Q ring attention to full prefill, persistent-KV prefill, and decode.
• Ring Attention with Blockwise Transformers - Earlier blockwise ring formulation for near-infinite-context training.

Related Terms

• Context Parallelism - Inference system strategy built around ring attention.
• Sequence Parallelism - Training strategy that uses Ring Self-Attention.
• All-to-All - Output redistribution required by pass-Q.