Knowledge note
Mixture of Experts
A sparse model architecture that routes each token through a small subset of many expert feed-forward networks to increase total capacity without activating every parameter.
Mixture of Experts
Mixture of Experts is a sparse neural-network architecture where a router sends each token to a small subset of many expert feed-forward networks, increasing total parameter capacity while keeping per-token active compute limited.
Why It Exists
Dense scaling activates every parameter for every token, so larger models cost proportionally more at training and inference. MoE separates total capacity from active compute: many experts store knowledge and specialization, but only a few are evaluated per token.
How It Works
A router scores experts for each token and selects a Top-k subset. The selected experts process the token representation, their outputs are weighted by router scores, and the aggregate returns to the main model stream. Large MoE systems usually combine routed experts with shared experts, expert parallelism, load-balancing rules, and communication overlap.
Tradeoffs
MoE introduces routing instability, expert load imbalance, dispatch/combination communication, and train–inference consistency issues. Extreme expert counts need explicit load-balancing mechanisms such as routing biases, quantile balancing, redundant experts, or expert-parallel planning.
Common Confusions
- Total parameters vs. active parameters: Total parameters count all experts; active parameters count only experts selected for one token.
- MoE vs. attention: MoE usually expands feed-forward/channel mixing capacity, while attention controls token mixing.
Where It Appears
- DeepSeek-V2 Multi-Head Latent Attention — Uses DeepSeekMoE alongside MLA to reduce active FFN compute.
- DeepSeek-V4 — Uses a 1.6T/284B-active MoE model with hybrid compressed attention.
- DeepSeek-V4.1-Flash — Uses a 552B multimodal MoE with 8B active parameters during prefill and 16B during decode.
- Kimi Linear — Evaluates KDA/MLA hybrid attention on a 48B MoE with 3B active parameters.
- Kimi K3 — Scales to 2.8T total parameters, 104B active, and 896 routed experts per layer.
- MiniMax GQA W4A4 Quantization Path — Traces quantized sparse-expert execution on GPU and Ascend NPU.
- vLLM Kimi K3 Code Reading — Maps Kimi K3’s routed experts into the vLLM runtime.
- vLLM-Ascend Architecture — Shows where Ascend-specific MoE kernels replace upstream execution.
- vLLM-Ascend Kimi K3 MoE Forward — Follows the full routed-expert forward path on Ascend.
- Attention Variants — Attention designs that reduce query/key redundancy or KV-cache pressure: multi-query, grouped-query, collaborative, and latent.
- DeepSeek-V3.2: Sparse Attention, Scaled RL, and Thinking in Tool-Use — DeepSeek-V3.2 introduces DeepSeek Sparse Attention (DSA) for sub-quadratic long-context efficiency, a scaled GRPO recipe with.
- DeepSeek V4 Attention: Code Reading Map — A navigable map of the DeepSeek V4 hybrid compressed attention implementation across vLLM (NVIDIA/AMD/XPU) and vllm-ascend.
- DSpark: Confidence-Scheduled Speculative Decoding — DeepSeek’s DSpark speculative decoding framework, combining semi-autoregressive draft generation with hardware-aware confidence.
- Triton in Practice: How vLLM and vllm-ascend Use Triton — A codebase-driven tour of Triton kernel development in vLLM (NVIDIA GPU) and vllm-ascend (Ascend NPU), covering infrastructure.
- vLLM — vLLM serving framework pages: PagedAttention paper, request-flow code learning path, continuous batching, and Kimi K3 code.
- DeepSeek-V4 Inference on Ascend: The DSA Serving Stack in vllm-ascend — How vllm-ascend runs DeepSeek-V4 end to end on Ascend NPUs: model override with mHC hyper-connections, hybrid c4/c128 compressor.
- vLLM Ascend — vLLM’s Ascend NPU port: code-reading notes and MoE forward implementation insights.
- Qwen3.5 / Qwen3.6 Inference Path on vLLM Ascend — A code-reading tour of the shared qwen3_5-family inference path: Qwen3.5-27B / Qwen3.6-27B (dense hybrid Mamba-Transformer.
- FlatQuant: Fast Learnable Affine Quantization — FlatQuant flattens outlier-heavy weights and activations with per-layer learnable affine transformations (Kronecker-factorized.
- NVFP4: Blackwell 4-Bit Floating Point — NVIDIA NVFP4 format with hierarchical FP8/FP32 scaling, micro-block quantization, Random Hadamard Transform, and Transformer.
- DeepGEMM MegaMoE: Fused Communication and Expert Compute — Shows how expert-parallel dispatch, persistent ring scheduling, fused expert MLP compute, and top-k combine overlap inside a Blackwell-oriented kernel.
- DeepSeek — DeepSeek model training papers: V4 hybrid compressed attention and V3.2 sparse attention with scaled RL.
- MiniMax Sparse Attention (MSA) — Blockwise sparse attention co-designed with GQA: a lightweight Index Branch selects top-k KV blocks per group, Main Branch.
- Kimi — Kimi model family: the Kimi Linear attention architecture and the Kimi K3 frontier model.
- mHC: Manifold-Constrained Hyper-Connections — DeepSeek’s mHC projects Hyper-Connections’ residual mixing matrix onto the doubly stochastic manifold with Sinkhorn-Knopp.
- vLLM DCP and PCP: Decode and Prefill Context Parallelism — PCP size participates in MoE sharding and process-world sizing.
Related Terms
- All-Gather — Common distributed primitive in parallel training systems.
- All-Reduce — Common gradient-synchronization primitive.