Why Do Current LLMs All Use Decoder-only Architectures?

为什么现在的LLM都是Decoder-only的架构？ 苏剑林 · 2023-03-17 · Machine Learning analysis language model text generation attention

LLM stands for "Large Language Model," and it currently generally refers to language models with more than ten billion parameters, mainly aimed at text generation tasks. Unlike the "let a hundred flowers bloom" diversity seen among smaller-scale models (on the order of a billion parameters or less), one notable feature of today's LLM landscape is that research on Decoder-only architectures dominates. Needless to say for OpenAI, which has consistently stuck with the Decoder-only GPT series—but even a company like Google, which hasn't bet everything on Decoder-only, has invested considerable effort into researching Decoder-only models, with PaLM being one example. So why has the Decoder-only architecture become the mainstream choice for LLMs?

There's a similar question on Zhihu, "Why do current LLMs all use Decoder-only architectures?", where most of the answers focus on the advantages of Decoder-only in terms of training efficiency and engineering implementation. But does it also have theoretical advantages? This post attempts a brief analysis from that angle.

A Unified Perspective

It should be noted that the largest models the author has personally trained top out at around the billion-parameter scale, so strictly speaking I'm not qualified to answer this question from the standpoint of "general" LLM knowledge. What follows is simply the author forcing an answer from a more theoretical angle, based on some research experience. Most of the inferences in this article are drawn from my own experimental results, and in some places they may conflict with the results of certain papers—readers are free to judge for themselves. more

We know that general NLP tasks predict an output based on a given input; fully unconditional random generation is rare. In other words, any NLP task can be decomposed into an "input" part and an "output" part. We can call the model that processes the "input" the Encoder, and the model that generates the "output" the Decoder. From this "Encoder-Decoder" perspective, all models can be understood, with the differences between models lying in the attention patterns of the Encoder and Decoder and whether they share parameters:

$$\begin{array}{c|ccc} \hline & \text{encoder attention} & \text{decoder attention} & \text{whether parameters are shared} \\ \hline \text{GPT} & \text{unidirectional} & \text{unidirectional} & \text{is} \\ \text{UniLM} & \text{bidirectional} & \text{unidirectional} & \text{is} \\ \text{T5} & \text{bidirectional} & \text{unidirectional} & \text{no} \\ \hline \end{array}$$

Here GPT is the representative work of Decoder-only; UniLM is a Decoder architecture similar to GPT, but with a mixed attention pattern; T5 is the representative work of the Encoder-Decoder architecture, which is mainly of interest to Google.

BidirectionalMixedUnidirectional (forward)Unidirectional (backward)

Google carried out fairly thorough comparative experiments in the papers T5 and UL2, and both results showed an advantage for the Encoder-Decoder architecture over Decoder-only. However, since from an LLM perspective the model scales in these two papers are still not particularly large, and since most LLMs today are indeed Decoder-only, there's still no answer to whether this advantage persists at larger scales, nor to what underlies the advantage itself.

Comparative Experiments

As we can see from the table above, GPT and UniLM are actually the pair that keeps the variables strictly controlled for comparison. If GPT is compared directly with T5, two variables actually change at once: the input attention becomes bidirectional, and the parameter count doubles. The reason these three are compared together is that their inference costs are roughly the same.

Compared with GPT, since T5 involves two changed variables, we cannot be sure whether the advantage of the Encoder-Decoder architecture mentioned above comes from switching the input attention to bidirectional, or from doubling the parameters. To settle this, the author ran a comparison between GPT and UniLM on a model at the billion-parameter scale. The results showed that, training from scratch with the same inputs and outputs (the loss is only computed on the output part in both cases, the only difference being the attention pattern on the input part), UniLM shows no advantage over GPT, and in fact performs worse on some tasks.

Assuming this conclusion is representative, we can tentatively draw the following conclusion:

Making the attention on the input bidirectional does not bring any benefit; the advantage of the Encoder-Decoder architecture is very likely simply due to the doubled parameter count.

In other words, given the same parameter count and the same inference cost, the Decoder-only architecture is very likely the optimal choice. Of course, to fully verify this hypothesis, some additional experiments are needed—for example, keeping the Encoder and Decoder parameters unshared but also making the Encoder's attention unidirectional, or switching it to the forward-backward mixed attention introduced in the next section, and then comparing against the conventional Encoder-Decoder architecture. But given the author's limited computational resources, these experiments are left to interested readers.

The Low-Rank Problem

Why does "making the input attention bidirectional bring no benefit"? Intuitively, since the input part doesn't need to worry about autoregressive generation, a full attention matrix should be better, right? The author suspects that this is very likely because the low-rank nature of bidirectional attention leads to a drop in performance.

As we know, the attention matrix is generally obtained by low-rank factorization followed by softmax—specifically, a $n\times d$ matrix multiplied by a $d\times n$ matrix, followed by softmax ($n\gg d$). This form of attention matrix suffers from reduced expressive power due to the low-rank issue; for a detailed analysis, see Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth. In contrast, the attention matrix of a Decoder-only architecture is a lower-triangular matrix, and note that the determinant of a triangular matrix equals the product of its diagonal entries. Because of softmax, the diagonal entries are necessarily all positive, so its determinant must be positive—meaning the attention matrix of a Decoder-only architecture is necessarily full rank! Full rank implies, in theory, stronger expressive power. That is, the attention matrix of the Decoder-only architecture theoretically has stronger expressive power, whereas switching to bidirectional attention actually becomes comparatively insufficient.

There's another phenomenon that indirectly supports this view: the gap between linear attention and standard attention on language modeling tasks (unidirectional attention) is smaller than the gap between them on MLM tasks (bidirectional attention). In other words, linear attention performs relatively worse on bidirectional-attention tasks. This is because, in language modeling tasks, linear attention's attention matrix is a full-rank lower-triangular matrix just like standard attention's; but in MLM tasks, the rank of the linear attention matrix is lower than that of the standard attention matrix (linear attention is a $n\times d$ matrix multiplied by a $d\times n$ matrix, so its rank cannot exceed $d$; standard attention is a $n\times d$ matrix multiplied by a $d\times n$ matrix followed by softmax, and softmax has some rank-boosting effect—see the "Low-Rank Problem" section and the comments in Transformer Upgrade Series: 3, From Performer to Linear Attention).

Conversely, can this conclusion be used to improve bidirectional-attention models like BERT? The idea isn't hard to come up with: for instance, in Multi-Head Attention, one half of the heads could have their attention matrices truncated to lower-triangular form (forward attention), while the other half are truncated to upper-triangular form (backward attention); or alternatively, odd layers could use lower-triangular truncation (forward attention) while even layers use upper-triangular truncation (backward attention). Both designs preserve the overall bidirectionality of interaction in the model as a whole (unlike GPT, where an earlier token can never interact with a later one), while also incorporating the full-rank advantage of unidirectional attention.

The author also ran a simple comparative experiment, and found that this forward-backward mixed attention performs slightly better on MLM tasks than a fully bidirectional attention model like BERT:

Comparison of training curves between full bidirectional attention and forward-backward mixed attention

The good news is that a slight advantage is visible, which indirectly supports the earlier hypothesis; the bad news is that this experiment only involves a base-scale (100-million-parameter) model, and the effect on larger models is still unclear.

Summary

So, the answer the author offers is this: the reason LLMs mainly use the Decoder-only architecture, besides its advantages in training efficiency and engineering implementation, is theoretically because the Encoder's bidirectional attention suffers from a low-rank problem, which may weaken the model's expressive power—and for generation tasks, introducing bidirectional attention brings no real benefit. As for why the Encoder-Decoder architecture performs better in certain scenarios, that's probably just because it has double the parameters. So, given the same parameter count and the same inference cost, the Decoder-only architecture is the optimal choice.