Knowledge note
DeepSeek
DeepSeek model training papers: V4 hybrid compressed attention and V3.2 sparse attention with scaled RL.
DeepSeek
- DeepSeek-V4: Million-Token Context via Hybrid Compressed Attention — 1.6T/284B MoE models with CSA+HCA hybrid attention, mHC, and Muon optimizer; 27% FLOPs and 10% KV cache of V3.2 at 1M-token contexts.
- DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression — 552B multimodal MoE with CED prefill, CSA2 cross-layer cache reuse, FP4 main KV, bounded SWA replay, and controllable reasoning effort.