Knowledge note
Efficient Attention Training
Training approaches built around efficient attention mechanisms: sparse, sliding-window, and delta-rule recurrent attention.
Efficient Attention Training
- IndexCache: Cross-Layer Index Reuse for Sparse Attention — Retains a minority of sparse-attention indexers and lets other layers reuse top-k selections; loss-guided search or multi-layer distillation preserves quality while removing up to 75% of indexer calls.
- MiniMax Sparse Attention (MSA) — Blockwise sparse attention co-designed with GQA: lightweight Index Branch selects top-k KV blocks per GQA group, Main Branch computes exact softmax attention over only the selected blocks, trained with KL alignment loss.
- SWAT: Sliding Window Attention Training — Sigmoid-based sliding window attention training: replaces softmax with sigmoid to eliminate attention sink, combines balanced bidirectional ALiBi with RoPE for training stability.
- Gated Delta Networks: Improving Mamba2 with Delta Rule — Fixed-state recurrent memory combining global adaptive decay with key-targeted correction (Delta Rule); decay-aware chunkwise WY training, and SWA/Mamba2 hybrids.
- YOCO: You Only Cache Once — Decoder-decoder architecture with a bounded-memory self-decoder, one shared global KV cache, causal cross-decoder layers, and chunk-parallel long-context training.
- Universal YOCO for Efficient Depth Scaling — Loops YOCO’s shallow efficient-attention self-decoder with shared parameters so recursion buys effective depth while the global KV cache is still written once; reports 41.78 → 47.08 average downstream scores, near-YOCO KV memory, and flat prefill throughput from 8K to 256K.