Knowledge note
Linear Attention
An attention family that factors query–key similarity through feature maps so key–value associations can be accumulated without an explicit quadratic attention matrix.
Linear Attention
Linear Attention is an attention family that factors query–key similarity through finite-dimensional feature maps, allowing values to be read from an accumulated key–value summary instead of an explicit token-by-token attention matrix.
Why It Exists
Softmax attention constructs an $N \times N$ score matrix, so its direct time and memory grow quadratically with sequence length. Autoregressive decoding also retains a history whose size and read cost grow with context. Linear attention seeks linear full-sequence cost and a fixed-size recurrent state.
How It Works
For a non-negative feature map $\phi$, associativity changes the evaluation order:
\[(\phi(Q)\phi(K)^T)V=\phi(Q)(\phi(K)^TV).\]The right side summarizes keys and values before applying queries. Under causal masking, prefix states $S_i=\sum_{j\le i}\phi(K_j)V_j^T$ (a $C \times M$ matrix) and $Z_i=\sum_{j\le i}\phi(K_j)$ (a $C$-vector) are updated once per token and queried as $\phi(Q_i)^TS_i/\phi(Q_i)^TZ_i$, where the numerator is an $M$-vector weighted value sum and the denominator is the scalar sum of the same similarity weights.
Modern variants add learned forgetting, delta-rule updates, convolution, or occasional full-attention layers to make the fixed-capacity state more selective.
How to Think About Queries, Keys, and Values
In both traditional and linear attention, $Q$, $K$, and $V$ come from the same token representation ($Q = XW_Q$, $K = XW_K$, $V = XW_V$) but serve different roles. The model learns $W_K$ and $W_V$ independently so addressing and content can specialize.
| Role | What it answers | Analogy |
|---|---|---|
| $K$ (key) | “How do I advertise this token so relevant queries find it?” | The address label — which folders to file in, and how strongly |
| $V$ (value) | “What useful information should this token contribute?” | The payload — the document being filed |
| $Q$ (query) | “Given my current context, which past tokens do I need?” | The search — which folders to read from, and how strongly |
The routing weight is always a $Q$–$K$ match, not $K$ alone: in traditional attention it’s $\text{softmax}(q_i^T k_j)$, and in linear attention it’s $\phi(Q_i)^T \phi(K_j)$. $K$ provides the address; $Q$ provides the search criteria.
The state matrix as a filing cabinet
$S_i = \sum_{j \le i} \phi(K_j) V_j^T$ is a $C \times M$ association table — rows are feature types, columns are value dimensions. Each new token adds a rank-1 contribution: $\phi(K_j)$ says how strongly this token belongs to each feature type, and $V_j$ is what content gets stored under those types. A query $\phi(Q_i)$ reads the cabinet by taking a weighted blend of rows.
This is the same filing-cabinet intuition as traditional attention — the difference is that linear attention compresses the cabinet first ($S_i$) and then queries it, while traditional attention flips through every individual folder ($k_j$) for every query.
Tradeoffs
Linear attention usually changes or approximates the softmax kernel, and its fixed-size state can blur distinct past tokens. It excels when context length dominates feature dimensions and exact random access is less important; hybrids retain full-attention layers for difficult retrieval cases.
Common Confusions
- Linear attention vs. FlashAttention: Linear attention changes or approximates the attention rule to avoid quadratic pairwise scores; FlashAttention computes exact softmax attention with IO-aware tiling.
- Linear attention vs. sparse attention: Linear attention compresses all history into a summary; sparse attention retains explicit access to a selected subset of tokens.
- Linear attention vs. an ordinary RNN: Causal linear attention has a recurrent matrix state, but keeps Transformer-style query, key, value projections and can use parallel prefix-style training.
Where It Appears
- Transformers Are RNNs: Linear Attention — Establishes kernel factorization, associative reordering, and the causal recurrent formulation.
- Linear Attention Without Softmax — Surveys kernel-map and related non-softmax formulations.
- Gated Delta Networks — Adds adaptive global decay and key-targeted delta updates to a matrix-valued recurrent state.
- Kimi Linear — Extends the family with channel-wise forgetting, delta-rule updates, and periodic full-attention layers.
- Kimi K3 — Scales Kimi-family linear attention to a 2.8T MoE with lower-bounded decay for BF16 Tensor Core kernels.
- vLLM-Ascend Prefill and Decode Scheduling: Qwen3.5 GQA — Contrasts Qwen3.5’s GDN recurrent layers with its full-attention GQA/FIA layers under one scheduler plan.
- Attention Foundations — Foundational sequence-modeling concepts behind attention-based LLMs: the Transformer, softmax, and RNN-to-LSTM background.
- Recurrent Neural Networks: From RNN to LSTM — A beginner-oriented explanation of recurrent hidden state, shared weights across sequence steps, long-term dependency failures.
- The Transformer: Attention Is All You Need — The foundational paper that introduced the Transformer architecture, dispensing with recurrence and convolutions entirely in.
- Triton in Practice: How vLLM and vllm-ascend Use Triton — A codebase-driven tour of Triton kernel development in vLLM (NVIDIA GPU) and vllm-ascend (Ascend NPU), covering infrastructure.
- vLLM Architecture and Code Organization Overview — A top-down code-reading map of the vLLM repository at commit a0c092ee72c0: how the V1 serving engine, model executor, config.
- vLLM Ascend — vLLM’s Ascend NPU port: code-reading notes and MoE forward implementation insights.
- Qwen3.5 / Qwen3.6 Inference Path on vLLM Ascend — A code-reading tour of the shared qwen3_5-family inference path: Qwen3.5-27B / Qwen3.6-27B (dense hybrid Mamba-Transformer.
- SWAT: Sliding Window Attention Training — Trains transformers from scratch with sigmoid-based sliding window attention, replacing softmax to eliminate attention sink and.
- Kimi — Kimi model family: the Kimi Linear attention architecture and the Kimi K3 frontier model.
Related Terms
- KV Cache — Explicit key/value history used by conventional autoregressive attention.
- Delta Rule — Key-targeted online error correction for associative memory.
- Kimi Delta Attention — A channel-wise gated delta-rule member of the linear-attention family.