FAQ on *"Why Are Current LLMs All Decoder-only Architectures?"*
Last week I wrote "Why Are Current LLMs All Decoder-only Architectures?"], summarizing some of my experimental findings and speculations on this topic. As expected for a hot topic, it got a lot of traffic — the repost on PaperWeekly passed ten thousand reads before long, and it also received quite a few upvotes on Zhihu. Across several platforms, I've been gradually receiving comments and questions from readers. I've collected some of the more representative ones into this FAQ, hoping it will help clear up further confusion.
Recap
In "Why Are Current LLMs All Decoder-only Architectures?"], I ran comparative experiments between the GPT and UniLM architectures, and combined with my past research experience, arrived at the following conjectures:
1. Making the attention in the input portion bidirectional does not bring any benefit; the advantage of the Encoder-Decoder architecture is most likely just due to the doubled parameter count.
2. Bidirectional attention brings no benefit, possibly because the low-rank issue of bidirectional attention causes performance to degrade.
Based on these two conjectures, we arrive at the conclusion:
Under equal parameter count and equal inference cost, the Decoder-only architecture is the optimal choice.
For the details of the relevant experiments and reasoning, please refer to the original post; I won't repeat them here.
Q&A
Here I give my own answers to some of the questions readers raised.
Question 1: $n \gg d$ doesn't seem to hold?
Answer: $n$ is the sequence length, and $d$ is the head_size, not the hidden_size. In multi-head attention, head_size = hidden_size / heads — for example, in BERT base, head_size = 768 / 12 = 64 — while the pretraining length $n$ is generally 512, so $n \gg d$ roughly holds in general.
Question 2: BERT and the original GPT have the same parameter count, so why is BERT better at understanding tasks?
Answer: BERT and GPT differ not only in architecture but also in pretraining task, so a fair comparison isn't possible. At the end of the original post, I already gave an idea for improving BERT using ideas from GPT, and preliminary experiments suggest it likely outperforms BERT — that experiment is the one that strictly controls for variables.
Question 3: "Performance degradation caused by the low-rank issue of bidirectional attention" sounds like it could be a bug. Most industrial models today use bidirectional attention — isn't the scope of this claim too broad?
Answer: I never claimed that "bidirectional attention is bad for any task whatsoever." The fact that "most industrial models today use bidirectional attention" doesn't actually conflict with the conclusion of the original post. Our experimental conclusion in the original post was that "introducing bidirectional attention into the Encoder for generation tasks doesn't seem to bring any benefit" — the condition of this conclusion is quite clear: it's specifically about "the Encoder in generation tasks."
Question 4: I don't buy it... Decoder models are just better suited for dialogue models. Inside Google, LLM-based encoder models, decoder models, and encoder-decoder models all exist, each suited to different scenarios; the other two perform better on other tasks.
Answer: The answer here is similar to the previous one. The phenomenon that "decoder models and encoder-decoder models both exist" doesn't contradict the conclusion of the original post. We only tentatively speculated that "introducing bidirectional attention into the Encoder for generation tasks doesn't seem to bring any benefit" — we never said that the doubled parameter count brought by the Encoder yields no benefit at all.
Question 5: Doesn't your conclusion contradict the conclusions of T5 and UL2?
Answer: First, the conclusion of the original post doesn't contradict UL2's. The original post conjectures that "under equal parameter count and equal inference cost, the Decoder-only architecture is optimal," while UL2's conclusion is that Encoder-Decoder performs better — but Encoder-Decoder and Decoder-only don't have equal parameter counts. Second, the conclusion of the original post does seem to conflict somewhat with the experimental results in T5 (Table 2). However, I also have doubts about T5's experimental results:
1. Whether the decoder-only and UniLM setups in that table really achieve strict control of variables — the gap between them is simply too large, and it feels unreasonable. Even if decoder-only is somewhat worse than UniLM, the gap shouldn't be that big.
2. In this post, the comparison is between UniLM and decoder-only trained from scratch under identical tasks and data (comparing pretraining results directly, without fine-tuning on other tasks), whereas the T5 paper compares results after pretraining on various tasks followed by fine-tuning on downstream tasks. Since the pipelines differ, could this account for the difference in outcomes?
Question 6: Does the fact that the loss drops faster in the final experiment really prove the model performs better?
Answer: Based on the number of training steps I've run so far, the hybrid (bidirectional + causal) attention setup has consistently performed better; I can only guess that this trend continues afterward — that's the limit of what I'm currently able to test experimentally. I look forward to readers with the interest and resources to run further experiments that could confirm or refute this conclusion.
Question 7: Regarding your statement that "GPT versus UniLM is the proper strict-variable-controlled comparison," I don't think that's quite accurate. Google's UL2 paper points out that, for pretrained language models, both model architecture and pretraining task play a crucial role in model quality.
Answer: In this post, "UniLM" and "GPT" refer to two model architectures that differ only in their attention mask. When running the comparative experiments, everything else besides the attention mask was kept aligned.
Question 8: Could there be another reason — that lower-triangular or upper-triangular masks handle positional encoding information better?
Answer: That's actually a quite novel viewpoint I hadn't considered before. But in fact, besides raising the rank, the triangular mask does indeed bring an advantage in position recognition as well — it breaks the permutation invariance of the Transformer and directly introduces a left-to-right ordering, so much so that it works even without positional encoding at all. Perhaps both factors are at play.
Summary
This post answered some of the questions readers raised about the previous article.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.