Why Does DeltaNet Add L2 Normalize?

为什么DeltaNet要加L2 Normalize？ 苏剑林 · 2025-12-23 · Mathematics differential equation linear RNN attention

In the article A Brief History of Linear Attention: From Imitation, to Innovation, to Feeding Back, we introduced DeltaNet, which brought the Delta Rule into linear attention, making it one of its more powerful tools, and laying the foundation for follow-up work such as GDN and KDA. However, that article focused mainly on the overall idea behind DeltaNet and didn't get into too many technical details — in this post we'll discuss one of them: why do DeltaNet and its follow-up works add L2 Normalize to $\boldsymbol{Q}、\boldsymbol{K}$?

Of course, explaining this operation directly from the perspective of eigenvalues isn't difficult, but I always felt something was missing. A few days ago I came across a new way of understanding it in the paper Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics, which I think has its merits, so I'd like to share it here.

The Basic Explanation

DeltaNet's recurrence is

\begin{equation}\boldsymbol{S}_t = \boldsymbol{S}_{t-1} - \eta_t (\boldsymbol{S}_{t-1} \boldsymbol{k}_t - \boldsymbol{v}_t)\boldsymbol{k}_t^{\top} = \boldsymbol{S}_{t-1}(\boldsymbol{I} - \eta_t \boldsymbol{k}_t\boldsymbol{k}_t^{\top}) + \eta_t \boldsymbol{v}_t \boldsymbol{k}_t^{\top}\label{eq:delta}\end{equation}

From the TTT perspective, this is online optimization of the loss $\frac{1}{2}\Vert\boldsymbol{S}\boldsymbol{k} - \boldsymbol{v}\Vert^2$ using the SGD optimizer with learning rate $\eta_t$ (the trainable parameter being $\boldsymbol{S}$). We know that optimizers tend to be sensitive to the learning rate, especially non-adaptive optimizers like SGD, and in DeltaNet this manifests as some extra requirements on the transition matrix $\boldsymbol{I} - \eta_t \boldsymbol{k}_t\boldsymbol{k}_t^{\top}$.

Specifically, since the transition matrices at different time steps are multiplied together during the recurrence, in order to avoid numerical blow-up, the transition matrix cannot have eigenvalues greater than 1 or less than -1. For the matrix $\boldsymbol{I} - \eta_t \boldsymbol{k}_t\boldsymbol{k}_t^{\top}$, one of its eigenvalues is $1 - \eta_t\Vert\boldsymbol{k}_t\Vert^2$ and the rest are all 1 (try proving this yourself), from which we get the constraint

\begin{equation}-1 \leq 1 - \eta_t\Vert\boldsymbol{k}_t\Vert^2 \leq 1\label{eq:cond}\end{equation}

To enforce this constraint, the common approach is to add L2 Normalize to $\boldsymbol{k}_t$ and a Sigmoid to $\eta_t$, so that all eigenvalues fall within $(0, 1]$ — and this is where the L2 Normalize on $\boldsymbol{K}$ comes from. As for the L2 Normalize on $\boldsymbol{Q}$, it isn't strictly necessary; it's mostly added "in passing" for the sake of symmetry, similar to the case of Short Conv, where adding Short Conv to $\boldsymbol{K}$ is actually the key part [reference].

Additional Remarks

By the way, for a long time people were accustomed to keeping all eigenvalues within $(0, 1]$, hence the choice of adding a Sigmoid to $\eta_t$. Later, Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues pointed out that negative eigenvalues can enhance DeltaNet's state-tracking ability, and proposed modifying DeltaNet into

\begin{equation}\boldsymbol{S}_t = \boldsymbol{S}_{t-1}(\boldsymbol{I} - 2\eta_t \boldsymbol{k}_t\boldsymbol{k}_t^{\top}) + \eta_t \boldsymbol{v}_t \boldsymbol{k}_t^{\top}\end{equation}

still adding L2 Normalize to $\boldsymbol{k}_t$ and a Sigmoid to $\eta_t$, so that the eigenvalue range of the transition matrix $\boldsymbol{I} - 2\eta_t \boldsymbol{k}_t\boldsymbol{k}_t^{\top}$ is expanded to $(-1, 1]$. However, state tracking is an ability that's more relevant to specialized syntax (such as code), so if we only train and test on natural language after this modification, we may not observe an obvious difference.

There's another detail worth noting: when $\eta_t=1$, the transition matrix $\boldsymbol{I} - 2\boldsymbol{k}_t\boldsymbol{k}_t^{\top}$ is an orthogonal matrix, which is fine in theory but not in practice, because for efficiency reasons we usually compute in at least BF16, and BF16's lower precision means the eigenvalues of $\boldsymbol{I} - 2\boldsymbol{k}_t\boldsymbol{k}_t^{\top}$ can occasionally fall below -1, still risking blow-up under long-term repeated multiplication. So we also need to keep $\eta_t$ from getting too close to 1.

In fact, the explanation above is already fairly complete and not particularly complicated, so my nitpicking is mostly a matter of personal taste: there isn't a unique way to enforce the condition $\eqref{eq:cond}$. For instance, one could instead introduce a Squash-like operation similar to Capsule, as in Longhorn, so we can't naturally derive L2 Normalize as the only solution — it's just one workable scheme among others.

A Continuous-Time Perspective

Now let's introduce the approach from the paper Error-Free Linear Attention is a Free Lunch: Exact Solution from Continuous-Time Dynamics, which I think offers a rather elegant derivation path — though of course that's a matter of taste too. It views equation $\eqref{eq:delta}$ as the Euler discretization, over the interval $[t-\eta_t, t]$, of the following differential equation:

\begin{equation}\frac{d}{dt}\boldsymbol{S}_t = \boldsymbol{S}_t\underbrace{(-\boldsymbol{k}_t\boldsymbol{k}_t^{\top})}_{\boldsymbol{A}_t} + \underbrace{\boldsymbol{v}_t \boldsymbol{k}_t^{\top}}_{\boldsymbol{B}_t}\label{eq:ode}\end{equation}

It then points out that the reason numerical blow-up occurs is that the discretization scheme isn't accurate enough, and proposes constructing the recurrence directly by solving the differential equation, rather than by approximate discretization. Since $\boldsymbol{A}_t$ and $\boldsymbol{B}_t$ are constants within the interval $[t-\eta_t,t]$, finding the recurrence from $t-\eta_t$ to $t$ amounts to solving a linear differential equation with constant coefficients, and the general result is

\begin{equation}\boldsymbol{S}_t = \boldsymbol{S}_{t-\eta_t} e^{\eta_t \boldsymbol{A}_t} + \boldsymbol{B}_t \boldsymbol{A}_t^{-1}(e^{\eta_t \boldsymbol{A}_t} - \boldsymbol{I})\label{eq:S-t-eta}\end{equation}

Switching the notation $\boldsymbol{S}_{t+\eta_t}$ back to $\boldsymbol{S}_{t-1}$, then substituting in the expression for $\boldsymbol{A}_t,\boldsymbol{B}_t$ and simplifying, we get

\begin{equation}\boldsymbol{S}_t = \boldsymbol{S}_{t-1} \left(\boldsymbol{I} - \frac{1 - e^{-\eta_t\Vert\boldsymbol{k}_t\Vert^2}}{\Vert\boldsymbol{k}_t\Vert^2}\boldsymbol{k}_t\boldsymbol{k}_t^{\top}\right) + \frac{1 - e^{-\eta_t\Vert\boldsymbol{k}_t\Vert^2}}{\Vert\boldsymbol{k}_t\Vert^2}\boldsymbol{v}_t \boldsymbol{k}_t^{\top}\label{eq:ode-deltanet}\end{equation}

This is the final result we're after. The original paper calls it "EFLA" (Error-Free Linear Attention). It's equivalent to replacing $\eta_t$ with $\frac{1 - e^{-\eta_t\Vert\boldsymbol{k}_t\Vert^2}}{\Vert\boldsymbol{k}_t\Vert^2}$, where $\Vert\boldsymbol{k}_t\Vert^2$ naturally appears in the denominator, and when multiplied by $\boldsymbol{k}_t\boldsymbol{k}_t^{\top}$, it effectively acts as an L2 Normalize on $\boldsymbol{K}$.

Mathematical Details

In the previous section we quickly presented the EFLA result, skipping over a lot of the mathematical details. In this section we'll fill in some of the discussion. Due to space constraints, we can only briefly touch on the key points of the derivation here, without going into full detail.

The core result of the previous section is equation $\eqref{eq:S-t-eta}$, the solution of the differential equation $d\boldsymbol{S}_t/dt=\boldsymbol{S}_t \boldsymbol{A} + \boldsymbol{B}$. To avoid confusion, we've dropped the subscript on $\boldsymbol{A},\boldsymbol{B}$ here, since within the interval of integration these quantities are indeed constant. If $\boldsymbol{B}=\boldsymbol{0}$, we can directly write down $\boldsymbol{S}_t=\boldsymbol{S}_0 e^{t\boldsymbol{A}}$, where $e^{t\boldsymbol{A}}$ is the matrix exponential; when $\boldsymbol{B}\neq \boldsymbol{0}$, we rewrite the equation as $d(\boldsymbol{S}_t + \boldsymbol{B}\boldsymbol{A}^{-1})/dt=(\boldsymbol{S}_t + \boldsymbol{B}\boldsymbol{A}^{-1})\boldsymbol{A}$, and then use the solution for the case $\boldsymbol{B}=\boldsymbol{0}$ to obtain

\begin{equation}\boldsymbol{S}_t = (\boldsymbol{S}_0 + \boldsymbol{B}\boldsymbol{A}^{-1})e^{t\boldsymbol{A}} - \boldsymbol{B}\boldsymbol{A}^{-1} = \boldsymbol{S}_0 e^{t\boldsymbol{A}} + \boldsymbol{B}\boldsymbol{A}^{-1}(e^{t\boldsymbol{A}} - \boldsymbol{I})\end{equation}

Finally, changing the starting point to $t-\eta_t$ and the endpoint to $t$, and restoring the subscript $t$ to $\boldsymbol{A},\boldsymbol{B}$, we arrive at equation $\eqref{eq:S-t-eta}$. Note that the last term involves the inverse matrix $\boldsymbol{A}^{-1}$, but in fact we don't need $\boldsymbol{A}$ to be invertible — it should be understood as expanding $(e^x-1)/x$ as a power series and then substituting into $x = \boldsymbol{A}$. Now let's focus again on equation $\eqref{eq:S-t-eta}$: for DeltaNet, $\boldsymbol{A}_t = -\boldsymbol{k}_t\boldsymbol{k}_t^{\top}$ is a rank-1 matrix, which allows for further simplification:

\begin{equation}f(\boldsymbol{x}\boldsymbol{y}^{\top}) = \sum_{n=0}^{\infty} a_n (\boldsymbol{x}\boldsymbol{y}^{\top})^n = a_0\boldsymbol{I} + \sum_{n=1}^{\infty} a_n (\boldsymbol{x}\boldsymbol{y}^{\top})^n = f(0)\boldsymbol{I} + \boldsymbol{x}\underbrace{\left(\sum_{n=1}^{\infty} a_n(\boldsymbol{y}^{\top}\boldsymbol{x})^{n-1}\right)}_{\frac{f(\boldsymbol{y}^{\top}\boldsymbol{x})-f(0)}{\boldsymbol{y}^{\top}\boldsymbol{x}}}\boldsymbol{y}^{\top}\end{equation}

Note that $\boldsymbol{y}^{\top}\boldsymbol{x}$ is a scalar, so the essence of this simplification is turning a matrix function into a scalar function, from which we obtain

\begin{equation}e^{\eta_t \boldsymbol{A}_t} = \boldsymbol{I} - \frac{1 - e^{-\eta_t\Vert\boldsymbol{k}_t\Vert^2}}{\Vert\boldsymbol{k}_t\Vert^2}\boldsymbol{k}_t\boldsymbol{k}_t^{\top},\qquad \boldsymbol{B}_t \boldsymbol{A}_t^{-1}(e^{\eta_t \boldsymbol{A}_t} - \boldsymbol{I})=\frac{1 - e^{-\eta_t\Vert\boldsymbol{k}_t\Vert^2}}{\Vert\boldsymbol{k}_t\Vert^2}\boldsymbol{v}_t \boldsymbol{k}_t^{\top}\end{equation}

Personal Reflections

That concludes our introduction to EFLA. The original paper also includes some experiments showing that EFLA has certain advantages over the original DeltaNet. But from equation $\eqref{eq:ode-deltanet}$ we can see that EFLA is still fundamentally of the same form as DeltaNet, so in principle we shouldn't expect it to bring about a "great leap forward." So why does EFLA generally perform slightly better? DeltaNet, through L2 Normalize, directly discards the magnitude of $\boldsymbol{K}$, whereas in equation $\eqref{eq:ode-deltanet}$, $\boldsymbol{v}_t \boldsymbol{k}_t^{\top}$ depends on $\Vert\boldsymbol{k}_t\Vert$ — so EFLA effectively has one more degree of freedom, which in theory gives it a somewhat higher ceiling.

Moreover, the idea of using the exact solution of a differential equation to construct a recurrence isn't new in EFLA. We mentioned this when discussing SSMs in Revisiting SSMs (II): Some Leftover Questions about HiPPO — the key result, equation $\eqref{eq:S-t-eta}$, had already appeared in HiPPO. What EFLA mainly does is carry out the expanded computation specifically for the special case of DeltaNet, arriving at a simplified, usable result.

A question worth thinking about more deeply is: what's the benefit of starting from a differential equation? It's not hard to see that the eigenvalues of the transition matrix in equation $\eqref{eq:ode-deltanet}$ automatically fall within $(0, 1]$ — that is, the recurrence obtained by solving the differential equation $\eqref{eq:ode}$ naturally has better stability. This is because a differential equation carries an implicit continuity constraint, and combined with the fact that matrix $-\boldsymbol{k}_t\boldsymbol{k}_t^{\top}$ is negative semi-definite, standard theory of differential equations tells us its solution is stable.

A classic example from mathematical modeling is the logistic equation $dx/dt = \alpha x - \beta x^2$, whose solution is simply the logistic function — very simple. But the corresponding difference equation $x_{t+1} - x_t = \alpha x_t - \beta x_t^2$ can exhibit chaotic behavior under certain settings (extreme sensitivity to initial conditions, to the point of being unpredictable). So, starting from a differential equation can automatically help us avoid such pathological behavior.

Summary

This article has discussed DeltaNet's L2 Normalize, focusing mainly on the idea of reparameterizing DeltaNet starting from a differential equation. It can also be seen as offering an explanation for the L2 Normalize operation applied to $\boldsymbol{K}$ in DeltaNet.