Knowledge note

Delta Rule

An online error-correction update that changes an associative memory in proportion to the difference between its current prediction and a target value.

Updated September 16, 2026Confidence: High

Delta Rule

Delta Rule is an online error-correction update that changes a model’s prediction at an input in proportion to the difference between its current prediction and the desired target.

Why It Exists

Simply adding key–value outer products makes a fixed-size associative memory accumulate interference. Global decay frees capacity but weakens every association together. The delta rule instead lets a new observation correct the memory specifically in the direction addressed by its key.

How It Works

For matrix memory $\mathbf S$, key $\mathbf k$, target value $\mathbf v$, and step size $\beta$, the update is

\[\mathbf S'=\mathbf S+\beta(\mathbf v-\mathbf S\mathbf k)\mathbf k^T.\]

First read the current prediction $\mathbf S\mathbf k$, subtract it from the target, and write that residual back along $\mathbf k$. DeltaNet interprets this as an online gradient step; Gated DeltaNet adds adaptive global decay, and Kimi Delta Attention makes that decay channel-wise.

Tradeoffs

The update is only as selective as the learned keys: overlapping key directions can interfere. Correcting one association at a time also does not rapidly clear a whole obsolete context, motivating a separate forgetting gate.

Common Confusions

  • Delta rule vs. global decay: Delta updates one key-aligned direction; decay scales the whole state.
  • Delta rule vs. Hebbian write: A Hebbian-style update adds $\mathbf v\mathbf k^T$ directly; the delta rule first subtracts the memory’s current prediction.

Where It Appears

  • Gated Delta Networks — Combines the delta rule with data-dependent scalar decay and a chunkwise parallel training algorithm.
  • Kimi Linear — Extends Gated DeltaNet to channel-wise decay under a hardware-efficient DPLR constraint.
  • Kimi K3 — Uses KDA’s delta-rule update inside the 3:1 KDA/MLA hybrid attention stack.
  • Triton in Practice: How vLLM and vllm-ascend Use Triton — A codebase-driven tour of Triton kernel development in vLLM (NVIDIA GPU) and vllm-ascend (Ascend NPU), covering infrastructure.
  • Efficient Attention Training — Training approaches built around efficient attention mechanisms: sparse, sliding-window, and delta-rule recurrent attention.