Guide to the Kimi DeltaNet Family of linear attention

Math notation

A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣q⟩\lvert q\rangle∣q⟩ is a column vector, ⟨k∣\langle k\rvert⟨k∣ is a row vector, ⟨k∣q⟩\langle k\rvert q\rangle⟨k∣q⟩ is a number, and ∣v⟩⟨k∣\lvert v\rangle\langle k\rvert∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.

Modern linear attention variants are complex, and a upon first glance it is not so easy to see what they are designed to achieve. For reference here is the state update equation for Kimi Delta Attention (KDA):

S~t=St−1Diag⁡(αt)\widetilde S_t = S_{t-1}\operatorname{Diag}(\alpha_t)St​=St−1​Diag(αt​) ∣v^t⟩=S~t∣kt⟩\lvert\widehat v_t\rangle = \widetilde S_t\lvert k_t\rangle∣vt​⟩=St​∣kt​⟩ ∣et⟩=βt(∣vt⟩−∣v^t⟩)\lvert e_t\rangle = \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right)∣et​⟩=βt​(∣vt​⟩−∣vt​⟩) St=S~t+∣et⟩⟨kt∣S_t = \widetilde S_t+\lvert e_t\rangle\langle k_t\rvertSt​=St​+∣et​⟩⟨kt​∣ ∣ot⟩=St(dk−1/2∣qt⟩)\lvert o_t\rangle = S_t\left(d_k^{-1/2}\lvert q_t\rangle\right)∣ot​⟩=St​(dk−1/2​∣qt​⟩)

The reason they are so difficult to understand is that this is the latest in a family of linear attention variants that have been developed over the last few years and the complexity of them has inevitably ballooned such that from the outside the latest variants appear inaccessible.

In this post we are going to walk through the DeltaNet family of linear attention variants, two of which are used by the latest Qwen and Kimi model families, and show how you might have arrived at the same equations by asserting simple things about your hidden state.

That is the route we will take:

softmax attention → linear attention → DeltaNet → Gated DeltaNet → KDA

Only after deriving KDA will we turn to the recurrent and chunkwise Triton programs that execute it.

1. Begin with quadratic attention

For a query at token ttt, ordinary causal softmax attention is

ati=exp⁡ ⁣(s⟨ki∣qt⟩)∑j≤texp⁡ ⁣(s⟨kj∣qt⟩),s=dk−1/2,∣ot⟩=∑i≤tati∣vi⟩.\begin{aligned} a_{ti} &= \frac{ \exp\!\left(s\langle k_i\rvert q_t\rangle\right) }{ \sum_{j\leq t} \exp\!\left(s\langle k_j\rvert q_t\rangle\right) }, \qquad s=d_k^{-1/2},\\ \lvert o_t\rangle &= \sum_{i\leq t}a_{ti}\lvert v_i\rangle. \end{aligned}ati​∣ot​⟩​=∑j≤t​exp(s⟨kj​∣qt​⟩)exp(s⟨ki​∣qt​⟩)​,s=dk−1/2​,=i≤t∑​ati​∣vi​⟩.​

Every attention weight is a scalar. It measures the similarity between one key and one query, then softmax turns all of the scores for that query into a distribution. The output is a weighted sum of value vectors.

Over a sequence of length TTT, there are T2T^2T2 key-query pairs. During autoregressive inference we can cache the keys and values instead of recomputing them, but the cache still grows with the sequence and every new query still has to inspect the entire history.

The obstacle to rearranging this computation is the softmax. Its denominator depends jointly on the current query and every earlier key. So, for the moment, remove it.

1.1 Remove the softmax

For clarity, absorb the constant scale sss into the query. The deliberately bare version of attention is then

∣ot⟩=∑i≤t⟨ki∣qt⟩∣vi⟩.\lvert o_t\rangle = \sum_{i\leq t} \langle k_i\rvert q_t\rangle \lvert v_i\rangle.∣ot​⟩=i≤t∑​⟨ki​∣qt​⟩∣vi​⟩.

The scalar inner product can move to the right:

∣ot⟩=∑i≤t∣vi⟩⟨ki∣qt⟩=(∑i≤t∣vi⟩⟨ki∣)∣qt⟩.\begin{aligned} \lvert o_t\rangle &= \sum_{i\leq t} \lvert v_i\rangle \langle k_i\rvert q_t\rangle\\ &= \left( \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert \right) \lvert q_t\rangle. \end{aligned}∣ot​⟩​=i≤t∑​∣vi​⟩⟨ki​∣qt​⟩=(i≤t∑​∣vi​⟩⟨ki​∣)∣qt​⟩.​

Everything that depends on the past can now be collected into one matrix of a fixed size V×KV \times KV×K:

St=∑i≤t∣vi⟩⟨ki∣\boxed{ S_t = \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert }St​=i≤t∑​∣vi​⟩⟨ki​∣​

and attention becomes a recurrent write followed by a read:

St=St−1+∣vt⟩⟨kt∣,∣ot⟩=St∣qt⟩.\boxed{ \begin{aligned} S_t &= S_{t-1} + \lvert v_t\rangle\langle k_t\rvert,\\ \lvert o_t\rangle &= S_t\lvert q_t\rangle. \end{aligned} }St​∣ot​⟩​=St−1​+∣vt​⟩⟨kt​∣,=St​∣qt​⟩.​​

The identity

(∣v⟩⟨k∣)∣q⟩=⟨k∣q⟩∣v⟩\left(\lvert v\rangle\langle k\rvert\right)\lvert q\rangle = \langle k\rvert q\rangle\lvert v\rangle(∣v⟩⟨k∣)∣q⟩=⟨k∣q⟩∣v⟩

is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state StS_tSt​.

This is linear in sequence length rather than quadratic: scan the tokens once, updating the same dv×dkd_v\times d_kdv​×dk​ state at every step. We have paid for that efficiency by discarding softmax’s normalization and selectivity. More sophisticated linear-attention methods use feature maps and normalizers, but this unadorned form exposes the memory problem that motivates DeltaNet.

1.2 Addition is not assignment

Suppose we write a pair ∣vt⟩⟨kt∣\lvert v_t\rangle\langle k_t\rvert∣vt​⟩⟨kt​∣ and immediately query the new state with that same key:

St∣kt⟩=(St−1+∣vt⟩⟨kt∣)∣kt⟩=St−1∣kt⟩+∣vt⟩⟨kt∣kt⟩⏟1=St−1∣kt⟩+∣vt⟩.\begin{aligned} S_t\lvert k_t\rangle &= \left( S_{t-1} + \lvert v_t\rangle\langle k_t\rvert \right) \lvert k_t\rangle\\ &= S_{t-1}\lvert k_t\rangle + \lvert v_t\rangle \underbrace{\langle k_t\rvert k_t\rangle}_{1}\\ &= S_{t-1}\lvert k_t\rangle+\lvert v_t\rangle. \end{aligned}St​∣kt​⟩​=(St−1​+∣vt​⟩⟨kt​∣)∣kt​⟩=St−1​∣kt​⟩+∣vt​⟩1⟨kt​∣kt​⟩​​=St−1​∣kt​⟩+∣vt​⟩.​

The write does not make the memory return ∣vt⟩\lvert v_t\rangle∣vt​⟩. It adds ∣vt⟩\lvert v_t\rangle∣vt​⟩ to whatever the memory already returned.

If the old state already produced the correct value, the additive write makes the new state produce twice that value. More generally, keys are not mutually orthogonal, so every write can interfere with previous writes. Linear attention has given us a compact associative memory, but its update behaves like += when what we want is closer to =.

1. DeltaNet: write the error, not the value

DeltaNet replaces the unconditional linear-attention write with a delta-rule correction. There are two useful ways to derive it.

2.1 Derivation one: demand that the write can be read back

Before writing token ttt, ask the memory what it currently associates with the new key:

∣v^t⟩=St−1∣kt⟩.\lvert\widehat v_t\rangle = S_{t-1}\lvert k_t\rangle.∣vt​⟩=St−1​∣kt​⟩.

If we want the memory to return ∣vt⟩\lvert v_t\rangle∣vt​⟩, we should not add the whole value. We should add only the difference:

∣vt⟩−∣v^t⟩.\lvert v_t\rangle-\lvert\widehat v_t\rangle.∣vt​⟩−∣vt​⟩.

Introduce a learned write strength βt∈[0,1]\beta_t\in[0,1]βt​∈[0,1] and define

∣et⟩=βt(∣vt⟩−St−1∣kt⟩).\lvert e_t\rangle = \beta_t \left( \lvert v_t\rangle - S_{t-1}\lvert k_t\rangle \right).∣et​⟩=βt​(∣vt​⟩−St−1​∣kt​⟩).

Then write this error at the current key:

St=St−1+∣et⟩⟨kt∣.\boxed{ S_t = S_{t-1} + \lvert e_t\rangle\langle k_t\rvert. }St​=St−1​+∣et​⟩⟨kt​∣.​

Now immediately read the same key:

St∣kt⟩=St−1∣kt⟩+∣et⟩⟨kt∣kt⟩=(1−βt)St−1∣kt⟩+βt∣vt⟩.\begin{aligned} S_t\lvert k_t\rangle &= S_{t-1}\lvert k_t\rangle + \lvert e_t\rangle \langle k_t\rvert k_t\rangle\\ &= (1-\beta_t)S_{t-1}\lvert k_t\rangle + \beta_t\lvert v_t\rangle. \end{aligned}St​∣kt​⟩​=St−1​∣kt​⟩+∣et​⟩⟨kt​∣kt​⟩=(1−βt​)St−1​∣kt​⟩+βt​∣vt​⟩.​

When βt=1\beta_t=1βt​=1, the result is exactly ∣vt⟩\lvert v_t\rangle∣vt​⟩. Smaller βt\beta_tβt​ moves the old prediction partway towards the target.

The correction is also local in key space. For any query ∣x⟩\lvert x\rangle∣x⟩ orthogonal to the current key,

⟨kt∣x⟩=0⟹(St−St−1)∣x⟩=∣et⟩⟨kt∣x⟩⏟0=0.\langle k_t\rvert x\rangle=0 \quad\Longrightarrow\quad (S_t-S_{t-1})\lvert x\rangle = \lvert e_t\rangle \underbrace{\langle k_t\rvert x\rangle}_{0} =0.⟨kt​∣x⟩=0⟹(St​−St−1​)∣x⟩=∣et​⟩0⟨kt​∣x⟩​​=0.

So the rank-one write changes the response in the selected key direction while leaving every orthogonal direction alone.

2.2 Derivation two: take one step on reconstruction loss

The same update falls out of an online learning objective. Treat the current key-value pair as one training example for the linear map SSS:

Lt(S)=12∥S∣kt⟩−∣vt⟩∥22.\mathcal L_t(S) = \frac12 \left\| S\lvert k_t\rangle-\lvert v_t\rangle \right\|_2^2.Lt​(S)=21​∥S∣kt​⟩−∣vt​⟩∥22​.

Its gradient with respect to the state is

∇SLt(S)=(S∣kt⟩−∣vt⟩)⟨kt∣.\nabla_S\mathcal L_t(S) = \left( S\lvert k_t\rangle-\lvert v_t\rangle \right) \langle k_t\rvert.∇S​Lt​(S)=(S∣kt​⟩−∣vt​⟩)⟨kt​∣.

This is visibly an outer product: a value-space prediction error times the key bra at which that error was observed. Take one gradient-descent step of size βt\beta_tβt​ from St−1S_{t-1}St−1​:

St=St−1−βt∇SLt(St−1)=St−1−βt(St−1∣kt⟩−∣vt⟩)⟨kt∣=St−1+βt(∣vt⟩−St−1∣kt⟩)⟨kt∣.\begin{aligned} S_t &= S_{t-1} - \beta_t\nabla_S\mathcal L_t(S_{t-1})\\ &= S_{t-1} - \beta_t \left( S_{t-1}\lvert k_t\rangle-\lvert v_t\rangle \right) \langle k_t\rvert\\ &= S_{t-1} + \beta_t \left( \lvert v_t\rangle-S_{t-1}\lvert k_t\rangle \right) \langle k_t\rvert. \end{aligned}St​​=St−1​−βt​∇S​Lt​(St−1​)=St−1​−βt​(St−1​∣kt​⟩−∣vt​⟩)⟨kt​∣=St−1​+βt​(∣vt​⟩−St−1​∣kt​⟩)⟨kt​∣.​

This is exactly the update we got by requiring immediate reconstruction. The two interpretations are the same:

as a memory operation, βt\beta_tβt​ controls how strongly to replace the old association;

as online learning, βt\beta_tβt​ is the step size;

as linear algebra, the change is a rank-one outer product.

2.3 The DeltaNet state transition

Expanding the error exposes DeltaNet as a structured state transition plus a new input:

St=St−1+βt(∣vt⟩−St−1∣kt⟩)⟨kt∣=St−1(I−βt∣kt⟩⟨kt∣)+βt∣vt⟩⟨kt∣.\begin{aligned} S_t &= S_{t-1} + \beta_t \left( \lvert v_t\rangle-S_{t-1}\lvert k_t\rangle \right) \langle k_t\rvert\\ &= S_{t-1} \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) + \beta_t\lvert v_t\rangle\langle k_t\rvert. \end{aligned}St​​=St−1​+βt​(∣vt​⟩−St−1​∣kt​⟩)⟨kt​∣=St−1​(I−βt​∣kt​⟩⟨kt​∣)+βt​∣vt​⟩⟨kt​∣.​

For a unit key, I−βt∣kt⟩⟨kt∣I-\beta_t\lvert k_t\rangle\langle k_t\rvertI−βt​∣kt​⟩⟨kt​∣ has eigenvalue 1−βt1-\beta_t1−βt​ in the current key direction and eigenvalue 111 in every orthogonal direction. It removes the old association along the current key before adding the new one.

DeltaNet fixes the write. It does not yet fix the lifetime of the state.

1. Gated DeltaNet: sometimes old information should disappear

The linear state compresses the whole history into one matrix. A read

St∣q⟩=∑i≤t⟨ki∣q⟩∣vi⟩S_t\lvert q\rangle = \sum_{i\leq t} \langle k_i\rvert q\rangle\lvert v_i\rangleSt​∣q⟩=i≤t∑​⟨ki​∣q⟩∣vi​⟩

cannot choose to skip an individual old token after that token has been folded into StS_tSt​. Every stored direction that overlaps the query contributes. The delta rule can correct the state around the current key, but stale information in other directions remains available and can distort future reads.

We therefore need a way to forget the old state before using it. Let αt∈[0,1]\alpha_t\in[0,1]αt​∈[0,1] be a learned scalar retention gate:

S~t=αtSt−1.\widetilde S_t = \alpha_t S_{t-1}.St​=αt​St−1​.

Run the same delta rule against this gated state:

S~t=αtSt−1,forget,∣v^t⟩=S~t∣kt⟩,predict,∣et⟩=βt(∣vt⟩−∣v^t⟩),correct,St=S~t+∣et⟩⟨kt∣,write.\boxed{ \begin{aligned} \widetilde S_t &= \alpha_tS_{t-1}, &&\text{forget},\\ \lvert\widehat v_t\rangle &= \widetilde S_t\lvert k_t\rangle, &&\text{predict},\\ \lvert e_t\rangle &= \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right), &&\text{correct},\\ S_t &= \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert, &&\text{write}. \end{aligned} }St​∣vt​⟩∣et​⟩St​​=αt​St−1​,=St​∣kt​⟩,=βt​(∣vt​⟩−∣vt​⟩),=St​+∣et​⟩⟨kt​∣,​​forget,predict,correct,write.​​

This is Gated Del

[truncated for AI cost control]