Attention Residuals Memoir

Attention Residuals 回忆录 苏剑林 · 2026-03-19 · Machine Learning model optimization attention scaling law depth

This post introduces one of our latest works Attention Residuals (AttnRes). As the name suggests, this uses the idea of Attention to improve Residuals.

Many readers have probably heard of the Pre Norm/Post Norm debate, but at the end of the day this is just an "internal squabble" within Residuals itself, and many of the later variations of Normalization follow the same pattern. A more interesting change was HC, which started down the path of enlarging the residual stream, but perhaps due to instability in its results, it didn't attract much attention. What happened afterward, many of you probably already know: at the end of last year, DeepSeek's mHC improved upon HC and validated its effectiveness at a larger scale.

Rather than further enlarging the residual stream, we chose a more radical route: directly using Attention between layers to replace Residuals. Of course, getting the whole pipeline to work required a lot of details and effort, so let me briefly recall the journey here.

AttnRes diagram more

Attention Between Layers

As usual, let's start with Residuals, which everyone should be quite familiar with by now. It takes the form

\begin{equation}\boldsymbol{x}_t = \boldsymbol{x}_{t-1} + \boldsymbol{f}_t(\boldsymbol{x}_{t-1})\end{equation}

Here we'll write it another way, which lets us see something deeper. Let $\boldsymbol{y}_t=\boldsymbol{f}_t(\boldsymbol{x}_{t-1})$, so that $\boldsymbol{x}_t=\boldsymbol{x}_{t-1}+\boldsymbol{y}_t$, and by convention $\boldsymbol{y}_0=\boldsymbol{x}_0$, from which it's easy to derive $\boldsymbol{x}_t=\boldsymbol{y}_0+\boldsymbol{y}_1+\cdots+\boldsymbol{y}_t$. It can then be equivalently written as

\begin{equation}\boldsymbol{y}_{t+1} = \boldsymbol{f}_{t+1}(\boldsymbol{y}_0+\boldsymbol{y}_1+\cdots+\boldsymbol{y}_t)\label{eq:res-sum}\end{equation}

That is, from the perspective of $\boldsymbol{y}$, Residuals takes the equally-weighted sum of $\boldsymbol{y}_0,\boldsymbol{y}_1,\cdots,\boldsymbol{y}_t$ as the input to $\boldsymbol{f}_{t+1}$ to obtain $\boldsymbol{y}_{t+1}$. A natural generalization then is to switch to a weighted sum:

\begin{equation}\boldsymbol{y}_{t+1} = \boldsymbol{f}_{t+1}\left(\sum_{s=0}^t a_{t+1,s}\boldsymbol{y}_s\right)\qquad \text{where}\qquad a_{t,s}\geq 0,\quad\sum_{s=0}^t a_{t+1,s}=1\label{eq:attnres-gen}\end{equation}

This is the seed of AttnRes. The expression above also imposes two additional constraints on $a_{t,s}$; let's first discuss why they're necessary:

1. The constraint $a_{t,s}\geq 0$ ensures that the same $\boldsymbol{y}_s$ always contributes in the same direction across different layers, avoiding the inconsistency where one layer wants to increase $\boldsymbol{y}_s$ while another wants to decrease $\boldsymbol{y}_s$. Intuitively, this should be friendlier to the model's learning process;
2. The $\boldsymbol{f}$ we use includes an In Norm, which normalizes the input first via $\newcommand{RMSNorm}{\mathop{\text{RMSNorm}}}\RMSNorm$. Since $\RMSNorm(\boldsymbol{x})=\RMSNorm(c\boldsymbol{x})$ holds identically for any $\forall c > 0$, weighted averaging and weighted summation are completely equivalent, so the constraint $\sum_{s=0}^t a_{t,s}=1$ doesn't reduce expressive power.

Hyper-Connections

Before diving into AttnRes, let's briefly review HC (Hyper-Connections), and show that it too can be understood as inter-layer Attention, which demonstrates that inter-layer Attention is indeed a more fundamental approach. HC changes Residuals to

\begin{equation}\boldsymbol{X}_t = \boldsymbol{H}_t^{res}\boldsymbol{X}_{t-1} + \boldsymbol{H}_t^{post} \boldsymbol{f}_t(\boldsymbol{H}_t^{pre}\boldsymbol{X}_{t-1})\end{equation}

where $\boldsymbol{X}\in\mathbb{R}^{k\times d},\boldsymbol{H}^{res}\in\mathbb{R}^{k\times k},\boldsymbol{H}^{pre}\in\mathbb{R}^{1\times k},\boldsymbol{H}^{post}\in\mathbb{R}^{k\times 1}$, with the classic choice being $k=4$. Simply put, the state variable is expanded by a factor of $k$; before feeding into $\boldsymbol{f}_t$, a $\boldsymbol{H}_t^{pre}$ matrix shrinks it back down by $1$; after the output, $\boldsymbol{H}_t^{post}$ expands it back up by $k$, and finally it's added to $\boldsymbol{H}_t^{res}$-modulated $\boldsymbol{x}_{t-1}$. If we don't restrict the form of $\boldsymbol{H}_t^{res},\boldsymbol{H}_t^{pre},\boldsymbol{H}_t^{post}$, then Post Norm, Highway, etc., are all special cases of HC.

Similarly, let $\boldsymbol{y}_t=\boldsymbol{f}_t(\boldsymbol{H}_t^{pre}\boldsymbol{X}_{t-1})$, so that $\boldsymbol{X}_t = \boldsymbol{H}_t^{res}\boldsymbol{X}_{t-1} + \boldsymbol{H}_t^{post} \boldsymbol{y}_t$, and by convention $\boldsymbol{X}_0 = \boldsymbol{H}_0^{post}\boldsymbol{y}_0$. This can also be expanded as $\boldsymbol{X}_t = \boldsymbol{H}_{t\leftarrow 1}^{res}\boldsymbol{H}_0^{post}\boldsymbol{y}_0 + \boldsymbol{H}_{t\leftarrow 2}^{res}\boldsymbol{H}_1^{post}\boldsymbol{y}_1 + \cdots + \boldsymbol{H}_{t\leftarrow t}^{res}\boldsymbol{H}_{t-1}^{post}\boldsymbol{y}_{t-1} + \boldsymbol{H}_t^{post}\boldsymbol{y}_t$, where $\boldsymbol{H}_{t\leftarrow s}^{res}$ is defined as $\boldsymbol{H}_t^{res}\boldsymbol{H}_{t-1}^{res}\cdots \boldsymbol{H}_{s+1}^{res}\boldsymbol{H}_s^{res}$. Further defining $\boldsymbol{H}_{t\leftarrow t+1}^{res} = \boldsymbol{I}$, we can write

\begin{equation}\boldsymbol{y}_{t+1} = \boldsymbol{f}_{t+1}(\boldsymbol{H}_{t+1}^{pre}\boldsymbol{x}_t) = \boldsymbol{f}_{t+1}\bigg(\sum_{s=0}^t\underbrace{\boldsymbol{H}_{t+1}^{pre}\boldsymbol{H}_{t\leftarrow s+1}^{res}\boldsymbol{H}_s^{post}}_{a_{t+1,s}}\boldsymbol{y}_s\bigg)\end{equation}

Note that every $\boldsymbol{H}_{t+1}^{pre}\boldsymbol{H}_{t\leftarrow s+1}^{res}\boldsymbol{H}_s^{post}$ is a $1\times 1$ matrix, equivalent to a scalar, so this is also the inter-layer Attention form of expression $\eqref{eq:attnres-gen}$. Readers familiar with linear attention should quickly recognize this result: HC is essentially a "rotated by 90 degrees" version of DeltaNet. In practice, the three $\boldsymbol{H}$ matrices are computed from simple linear layers with $\tanh$ activation, which means the chained product of $\boldsymbol{H}_{t\leftarrow s}^{res}$ risks exploding or collapsing, and there's also no guarantee that $a_{t+1,s}$ is non-negative.

Later, mHC made an improvement: it first changed all three $\boldsymbol{H}$ to Sigmoid activations, guaranteeing non-negativity of $a_{t+1,s}$, then applied alternating normalization to $\boldsymbol{H}_t^{res}$ to make it doubly stochastic. The closure of doubly stochastic matrices under multiplication ensures the stability of $\boldsymbol{H}_{t\leftarrow s}^{res}$, and experiments confirmed the effectiveness of these changes. That said, some newer experiments, such as Maybe Your DeepSeek mHC Doesn't Need the "m", show that simply setting $\boldsymbol{H}_t^{res}$ to the identity matrix is already good enough.

Many Hands Make Light Work

Let's return to AttnRes. Once we realized AttnRes was feasible, the next question was: what form should $a_{t+1,s}$ take? A very natural idea is to follow the standard Scaled Dot-Product Attention, but at the time I wanted to try something quick first, so I picked a simpler form:

\begin{equation}a_{t+1,s} \propto \exp(\boldsymbol{w}_{t+1}\cdot \boldsymbol{y}_s)\end{equation}

where $\boldsymbol{w}_t$ is a trainable vector parameter — that is, using a data-independent static vector directly as Q, while both K and V are computed from $\boldsymbol{y}_s$ via Softmax Attention. This was the first version of AttnRes. Delightfully, even with such a simple design, the improvement over Residuals was already quite substantial!

After I shared the preliminary experimental results of AttnRes within the team, @Zhang Yu and @Guang Yu showed great interest and joined in, and we started running validation at a larger model scale, finding the results consistently encouraging. Along the way, we also tried some more complex designs, but found most of them underperformed relative to this simple version — only adding a $\RMSNorm$ operation to K yielded a fairly stable gain, which forms the final version of AttnRes:

\begin{equation}a_{t+1,s} \propto \exp(\boldsymbol{w}_{t+1}\cdot \RMSNorm(\boldsymbol{y}_s))\end{equation}

However, AttnRes is, after all, a densely connected inter-layer scheme — was training and inference at K2 scale or even larger actually feasible? Encouragingly, @V-ge, through careful analysis, first confirmed feasibility for inference, and the key insight was precisely the static-Q design we'd adopted for convenience at the start! This means that once we've computed $\boldsymbol{y}_s$, we can precompute the attention $a_{t,s}$ over $t > s$ ahead of time, which gave Infra enough room to maneuver.

Unfortunately, however, our training colleagues, such as @Wang-ge, after careful analysis, determined that under our current training setup, AttnRes still wasn't quite feasible (frankly, it came down to being resource-constrained), and we needed a scheme that further reduced communication and memory overhead. This led to the Block version below; correspondingly, the earlier version we now call the Full version.

The Block Version

Going from Full AttnRes to Block AttnRes is analogous to the historical process of linearizing quadratic Attention — all the existing ideas for Efficient Attention can be tried here. For example, the first thing we tried was SWA (Sliding Window Attention), but the actual results turned out quite poor, even worse than Residuals.

Upon reflection, I think this can be understood as follows: Residuals is itself already an extremely strong baseline, corresponding to an equally-weighted sum over all state vectors. For any new design to surpass it, it must at least be able to subsume it as a special case. Full AttnRes clearly satisfies this condition, but adding SWA on top does not — it discards part of the states, and thus cannot cover the special case of "equally-weighted sum over all state vectors."

This made us realize that, for AttnRes, "compression" might be more effective than "sparsification," and the compression need not be very fine-grained — a simple weighted sum might suffice. After some further thought and refinement, @Zhang Yu and @Guang Yu proposed the Block AttnRes design described in the paper, which combines block-wise processing with summation-based compression, achieving results close to the Full version.

The idea behind Block AttnRes is roughly this: first, the Embedding layer is treated as its own block on its own, because by observing the Attention matrix of the Full version (this is one of the benefits of the Attention framing — you can visualize the attention pattern at any time), we found that the model tends to allocate a considerable amount of attention to the Embedding layer, so it makes sense to separate it out. The remaining layers are grouped into blocks of $m$ layers each; within each block, summation is used for compression, and Attention across blocks is computed using these summed representations as units.

Experiments show that with roughly 8 blocks, we can already capture most of AttnRes's benefit. After evaluation, both the training and inference teams agreed that the extra overhead of Block AttnRes was small enough to be well worth the improvement (for detailed analysis, see @Wang-ge's and @V-ge's posts; roughly speaking, it's under 5% overhead in exchange for a 25% gain). So everyone pushed hard to get it into the main line — another rich and enjoyable experience that I won't elaborate on further here.

A Matrix Perspective

It's worth noting that, via the Attention matrix, we can unify Residuals, HC/mHC, Full AttnRes, and Block AttnRes into a single framework — an interesting way to think about it, illustrated below. Here $\phi(\boldsymbol{q},\boldsymbol{k}) = \exp(\boldsymbol{q}\cdot \RMSNorm(\boldsymbol{k}))$, the Block AttnRes version corresponds to $m=3$, and $\boldsymbol{y}_{s:t}=\sum_{i=s}^t \boldsymbol{y}_i$; this latter notation is one we also used in Making Model Training More Scientific (IV): New Identities, New Learning Rates.

Residuals

$$\boldsymbol{A}=\left(\begin{array}{c} 1 \\ 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \end{array}\right)$$

HC/mHC

$$\boldsymbol{A}=\left(\begin{array}{c} \boldsymbol{H}_1^{pre} \boldsymbol{H}_0^{post} \\ \boldsymbol{H}_2^{pre}\boldsymbol{H}_{1\leftarrow 1}^{res}\boldsymbol{H}_0^{post} & \boldsymbol{H}_2^{pre}\boldsymbol{H}_1^{post} \\ \boldsymbol{H}_3^{pre}\boldsymbol{H}_{2\leftarrow 1}^{res}\boldsymbol{H}_0^{post} & \boldsymbol{H}_3^{pre}\boldsymbol{H}_{2\leftarrow 2}^{res}\boldsymbol{H}_1^{post} & \boldsymbol{H}_3^{pre}\boldsymbol{H}_2^{post} \\ \boldsymbol{H}_4^{pre}\boldsymbol{H}_{3\leftarrow 1}^{res}\boldsymbol{H}_0^{post} & \boldsymbol{H}_4^{pre}\boldsymbol{H}_{3\leftarrow 2}^{res}\boldsymbol{H}_1^{post} & \boldsymbol{H}_4^{pre}\boldsymbol{H}_{3\leftarrow 3}^{res}\boldsymbol{H}_2^{post} & \boldsymbol{H}_4^{pre}\boldsymbol{H}_3^{post} \\ \boldsymbol{H}_5^{pre}\boldsymbol{H}_{4\leftarrow 1}^{res}\boldsymbol{H}_0^{post} & \boldsymbol{H}_5^{pre}\boldsymbol{H}_{4\leftarrow 2}^{res}\boldsymbol{H}_1^{post} & \boldsymbol{H}_5^{pre}\boldsymbol{H}_{4\leftarrow 3}^{res}\boldsymbol{H}_2^{post} & \boldsymbol{H}_5^{pre}\boldsymbol{H}_{4\leftarrow 4}^{res}\boldsymbol{H}_3^{post} & \boldsymbol{H}_5^{pre}\boldsymbol{H}_4^{post} \\ \boldsymbol{H}_6^{pre}\boldsymbol{H}_{5\leftarrow 1}^{res}\boldsymbol{H}_0^{post} & \boldsymbol{H}_6^{pre}\boldsymbol{H}_{5\leftarrow 2}^{res}\boldsymbol{H}_1^{post} & \boldsymbol{H}_6^{pre}\boldsymbol{H}_{5\leftarrow 3}^{res}\boldsymbol{H}_2^{post} & \boldsymbol{H}_6^{pre}\boldsymbol{H}_{5\leftarrow 4}^{res}\boldsymbol{H}_3^{post} & \boldsymbol{H}_6^{pre}\boldsymbol{H}_{5\leftarrow 4}^{res}\boldsymbol{H}_4^{post} & \boldsymbol{H}_6^{pre}\boldsymbol{H}_5^{post} \\ \boldsymbol{H}_7^{pre}\boldsymbol{H}_{6\leftarrow 1}^{res}\boldsymbol{H}_0^{post} & \boldsymbol{H}_7^{pre}\boldsymbol{H}_{6\leftarrow 2}^{res}\boldsymbol{H}_1^{post} & \boldsymbol{H}_7^{pre}\boldsymbol{H}_{6\leftarrow 3}^{res}\boldsymbol{H}_2^{post} & \boldsymbol{H}_7^{pre}\boldsymbol{H}_{6\leftarrow 4}^{res}\boldsymbol{H}_3^{post} & \boldsymbol{H}_7^{pre}\boldsymbol{H}_{6\leftarrow 5}^{res}\boldsymbol{H}_4^{post} & \boldsymbol{H}_7^{pre}\boldsymbol{H}_{6\leftarrow 6}^{res}\boldsymbol{H}_5^{post} & \boldsymbol{H}_7^{pre}\boldsymbol{H}_6^{post} \\ \end{array}\right)$$

Full AttnRes

$$\boldsymbol{A}=\left(\begin{array}{c} \phi(\boldsymbol{w}_1, \boldsymbol{y}_0) \\ \phi(\boldsymbol{w}_2, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_2, \boldsymbol{y}_1) \\ \phi(\boldsymbol{w}_3, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_3, \boldsymbol{y}_1) & \phi(\boldsymbol{w}_3, \boldsymbol{y}_2) \\ \phi(\boldsymbol{w}_4, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_4, \boldsymbol{y}_1) & \phi(\boldsymbol{w}_4, \boldsymbol{y}_2) & \phi(\boldsymbol{w}_4, \boldsymbol{y}_3) \\ \phi(\boldsymbol{w}_5, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_1) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_2) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_3) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_4) \\ \phi(\boldsymbol{w}_6, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_1) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_2) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_3) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_4) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_5) \\ \phi(\boldsymbol{w}_7, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_1) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_2) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_3) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_4) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_5) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_6) \\ \end{array}\right)$$

Block AttnRes

$$\boldsymbol{A}=\left(\begin{array}{c:ccc:ccc} \phi(\boldsymbol{w}_1, \boldsymbol{y}_0) \\ \hdashline \phi(\boldsymbol{w}_2, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_2, \boldsymbol{y}_1) \\ \phi(\boldsymbol{w}_3, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_3, \boldsymbol{y}_{1:2}) & \phi(\boldsymbol{w}_3, \boldsymbol{y}_{1:2}) \\ \phi(\boldsymbol{w}_4, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_4, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_4, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_4, \boldsymbol{y}_{1:3}) \\ \hdashline \phi(\boldsymbol{w}_5, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_5, \boldsymbol{y}_4)\\ \phi(\boldsymbol{w}_6, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_{4:5}) & \phi(\boldsymbol{w}_6, \boldsymbol{y}_{4:5}) \\ \phi(\boldsymbol{w}_7, \boldsymbol{y}_0) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_{1:3}) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_{4:6}) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_{4:6}) & \phi(\boldsymbol{w}_7, \boldsymbol{y}_{4:6}) \\ \end{array}\right)$$

Related Work

Ever since we set out to work on AttnRes, my colleagues and I have been fully immersed in the process of refining, validating, and accelerating it. Some readers may know that my research style is to push as far as I can on my own with derivation and problem-solving, and only look into the related literature once I hit a wall or have fully worked things out. As it happened, I found myself among a group of like-minded collaborators, and this particular exploration of AttnRes went smoothly overall — so it wasn't until all the tests had basically passed and we started preparing the technical report that we began surveying the related literature.

But precisely because of that, "you don't know until you look" — it turns out there's already a large body of work on dense connections and depth attention. Besides the classic DenseNet, we also found DenseFormer, ANCRe, MUDDFormer, MRLA, Dreamer, and others — even ELMo, which predates BERT, partially applied a similar design. We've included all of these in our references.

After the technical report went out, we gradually received comments from readers pointing out related works we hadn't yet included, such as SKNets, LIMe, DCA, and others. We apologize for the omissions and are grateful for the pointers, and we promise to add them in a future revision as much as possible. But whether you're a reader or the author, please keep a level head about this — a literature review is not an easy task, and some omissions are inevitable. We hold all related works in high regard.

At the same time, we'd also like to draw attention to the amount of engineering work involved in AttnRes beyond the "Depth Attention" concept itself. We fully agree that, here in 2026, "Depth Attention" or "Layer Attention" is by itself a fairly unoriginal idea. But making it work for models large enough, as a genuinely strong substitute for Residuals, while also meeting the efficiency requirements of both training and inference, is by no means easy. As far as we know, AttnRes is the first piece of work to achieve this.

Summary

This post introduced our latest architectural result, Attention Residuals (AttnRes), which replaces plain Residuals with inter-layer Attention. Through careful design, it meets the efficiency requirements of both training and inference, and we ultimately succeeded in scaling it up to sufficiently large models.