Why Does Pre-Norm Underperform Post-Norm?
The comparison between Pre Norm and Post Norm is a well-worn topic, and this blog has discussed it several times before, e.g. in A Brief Discussion on Transformer Initialization, Parameterization, and Normalization and Musings on Model Optimization: Why Is BERT's Initial Standard Deviation 0.02?, among others. The fairly settled conclusion at this point is: under the same setup, the Pre Norm structure tends to be easier to train, but its final performance is usually worse than Post Norm. It's easy to understand why Pre Norm is easier to train—its identity path is more prominent—but why does it end up performing worse?
I never had a satisfying answer to this myself, until some time ago I came across a reply on Zhihu by @唐翔昊, which was a genuine "aha" moment—it turns out there's a wonderfully intuitive way to understand this issue! Let's go through it together in this post. more
The Basic Conclusion
The formulas for Pre Norm and Post Norm are as follows:
\begin{align} \text{Pre Norm: } \quad \boldsymbol{x}_{t+1} = \boldsymbol{x}_t + F_t(\text{Norm}(\boldsymbol{x}_t))\\ \text{Post Norm: }\quad \boldsymbol{x}_{t+1} = \text{Norm}(\boldsymbol{x}_t + F_t(\boldsymbol{x}_t)) \end{align}
In Transformers, $\text{Norm}$ here mainly refers to Layer Normalization, but in more general models it could also be Batch Normalization, Instance Normalization, etc. — the conclusions below are essentially universal.
Among the sources I've found, there are two pieces of work showing that Post Norm outperforms Pre Norm: one is Understanding the Difficulty of Training Transformers, and the other is RealFormer: Transformer Likes Residual Attention. In addition, I've run my own comparison experiments, which show that the Post Norm structure has better transfer performance—that is, during pretraining, Pre Norm and Post Norm can reach roughly the same results, but Post Norm's fine-tuning performance is noticeably better.
Readers might push back and ask: doesn't On Layer Normalization in the Transformer Architecture show that Pre Norm is better than Post Norm? Isn't that a contradiction? Actually, that paper compares Pre Norm and Post Norm under exactly the same training setup, and shows only that Pre Norm is easier to train under those conditions. This is because Post Norm needs a different training configuration to reach its own optimum (for instance, Pre Norm can skip warmup, but Post Norm usually needs it). So there's no real contradiction between the two findings.
An Intuitive Understanding
Why does Pre Norm underperform Post Norm? The answer given by @唐翔昊 on Zhihu is: Pre Norm's depth is "watered down"! In other words, an $L$-layer Pre Norm model is not actually as deep, in effective terms, as an $L$-layer Post Norm model — and having fewer effective layers is what hurts performance.
How exactly should we understand this? It's actually quite simple. For the Pre Norm model, iterating gives us:
\begin{equation}\begin{aligned} \boldsymbol{x}_{t+1} =&\,\boldsymbol{x}_t + F_t(\text{Norm}(\boldsymbol{x}_t)) \\ =&\, \boldsymbol{x}_{t-1} + F_{t-1}(\text{Norm}(\boldsymbol{x}_{t-1})) + F_t(\text{Norm}(\boldsymbol{x}_t)) \\ =&\, \cdots \\ =&\, \boldsymbol{x}_0 + F_0 (\text{Norm}(\boldsymbol{x}_0)) + \cdots + F_{t-1}(\text{Norm}(\boldsymbol{x}_{t-1})) + F_t(\text{Norm}(\boldsymbol{x}_t)) \end{aligned}\end{equation}
Since each term here is of the same order of magnitude, we have $\boldsymbol{x}_{t+1}=\mathcal{O}(t+1)$, meaning that the difference between layer $t+1$ and layer $t$ is comparable to the difference between $t+1$ and $t$. When $t$ is large, this relative difference becomes very small, so
\begin{equation}\begin{aligned} &\,F_t(\text{Norm}(\boldsymbol{x}_t)) + F_{t+1}(\text{Norm}(\boldsymbol{x}_{t+1})) \\ \approx&\,F_t(\text{Norm}(\boldsymbol{x}_t)) + F_{t+1}(\text{Norm}(\boldsymbol{x}_t)) \\ =&\, \begin{pmatrix} 1 & 1\end{pmatrix}\begin{pmatrix} F_t \\ F_{t+1}\end{pmatrix}(\text{Norm}(\boldsymbol{x}_t)) \end{aligned}\end{equation}
This says that when $t$ is fairly large, the difference between $\boldsymbol{x}_t,\boldsymbol{x}_{t+1}$ is small, so $F_{t+1}(\text{Norm}(\boldsymbol{x}_{t+1}))$ and $F_{t+1}(\text{Norm}(\boldsymbol{x}_t))$ end up very close to each other. As a result, what was originally an $t$-layer model plus $t+1$ more layers ends up behaving approximately like a single, wider $t$-layer model. In other words, stacking more layers in Pre Norm mostly increases width rather than depth — the more layers you add, the more "hollow" each additional layer becomes.
Put plainly, the Pre Norm structure implicitly increases the model's width while reducing its effective depth, and we know that depth is generally more important than width — so this hidden reduction in depth is what causes the final performance to suffer. Post Norm works the opposite way: as we analyzed in A Brief Discussion on Transformer Initialization, Parameterization, and Normalization, each normalization step weakens the identity branch a little, so Post Norm actually emphasizes the residual branch more. As a result, the layers in Post Norm carry their "full weight," and once properly trained, it tends to achieve better final performance.
Related Work
Many readers have probably heard of DeepNet, which made headlines a while back for claiming to train 1000-layer Transformers. In its paper, DeepNet: Scaling Transformers to 1,000 Layers, the description of Pre Norm reads:
However, the gradients of Pre-LN at bottom layers tend to be larger than at top layers, leading to a degradation in performance compared with Post-LN.
Many readers may not have fully grasped the logic behind this statement at the time, but after reading the explanation in the previous section, it should make more sense now.
Simply put, "the gradients of Pre-LN at bottom layers tend to be larger than at top layers" means that the Pre Norm structure leans too heavily toward the identity branch (the bottom layers), causing Pre Norm to effectively degenerate into a "shallow but wide" model — which ultimately falls short of a Post Norm model of the same depth. This is fundamentally the same as the intuitive picture described above.
Summary
This post has shared an intuitive way of understanding why Pre Norm underperforms Post Norm.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.