Before Using ALBERT and ELECTRA, Make Sure You Really Understand Them

用ALBERT和ELECTRA之前，请确认你真的了解它们 苏剑林 · 2020-10-29 · Machine Learning language model attention

Among pretrained language models, ALBERT and ELECTRA are two "rising stars" that came after BERT. They each improved on BERT from different angles, ultimately boosting performance (at least on quite a few public benchmark datasets), and thereby earning a certain amount of goodwill. But in day-to-day discussions and study, I've noticed that many people harbor some misconceptions about these two models, to the point of wasting unnecessary time in using them. Here, I try to summarize some key points about these two models for reference, hoping to help everyone avoid some detours when using them.

ALBERT and ELECTRA

(Note: In this article, "BERT" refers both to the originally released BERT model and to the later improved version, RoBERTa. We can think of BERT as an undertrained RoBERTa, and RoBERTa as a more thoroughly trained BERT. This article is mainly concerned with comparing it against ALBERT and ELECTRA, so we won't distinguish between BERT and RoBERTa.)more

ALBERT

ALBERT comes from the paper ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. As the name suggests, it considers its own defining trait to be "Lite" — but what exactly does "Lite" mean here? Many domestic readers' impression of ALBERT is that it's small, fast, and good all at once. Is that really the case?

Characteristics

Simply put, ALBERT is essentially a parameter-shared BERT: it changes the function $y=f_n(f_{n-1}(\cdots(f_1(x))))$ into $y=f(f(\cdots(f(x))))$, where $f$ represents each layer of the model. So whereas there used to be $n$ layers' worth of parameters, now there's only one layer's worth, greatly reducing the parameter count — or in other words, the saved model weights take up much less disk space. This is the first meaning of "Lite." Then, since the total number of parameters is reduced, the time and GPU memory required for training also decrease correspondingly — this is the second meaning of "Lite." Furthermore, when the model is large, parameter sharing acts as a strong regularizer, so compared to BERT it is less prone to overfitting, and this ultimately gives its large-scale models a performance boost. This is ALBERT's real highlight.

Inference

Note that we haven't mentioned inference speed. Clearly, parameter sharing brings no speedup at inference time, since the model still just does forward computation step by step — it doesn't care whether the current parameters are the same as the previous ones, and even if they were the same it couldn't be sped up (because the input is different anyway). So an ALBERT and a BERT of the same size have the same inference speed — in fact, if we're being really strict about it, ALBERT should actually be slightly slower, because ALBERT applies matrix factorization to the embedding layer, which introduces some extra computation, even though this computational overhead is usually imperceptible.

Training

As for training speed, while there is some improvement, it's nowhere near as dramatic as one might imagine. Shrinking the parameter count to $1/n$ of the original does not mean training speed increases by a factor of $n$. In my own earlier experiments, the base version of ALBERT was only about 10%–20% faster to train than the base version of BERT, with a similar reduction in GPU memory usage. If the model is even smaller (tiny/small versions), this gap narrows further. In other words, ALBERT's training advantage is only noticeable for large models; for smaller models, this advantage remains hard to perceive.

Performance

As for performance, ALBERT's original paper already makes this quite clear, as shown in the table below. Parameter sharing limits the model's expressive capacity, so ALBERT's xlarge version is only able to match BERT's large version, and to reliably surpass it requires the xxlarge version. In other words, as long as the model size is smaller than xlarge, ALBERT's performance at that size is inferior to BERT's. Evaluation results on Chinese tasks show a similar pattern; see here and here. I also once ran an even more extreme experiment: loading ALBERT's weights but lifting the parameter-sharing constraint, effectively using ALBERT as a BERT — and performance improved! (See Discarding the Constraint, Strengthening the Model: Improving ALBERT's Performance with One Line of Code.) So it's basically confirmed: small-size ALBERT is inferior to BERT.

ALBERT experimental results

Conclusion

So, to summarize the recommendation: unless you're using the xlarge version, there's no need to use ALBERT — an ALBERT with the same speed as a given BERT performs worse, and an ALBERT with the same performance as a given BERT is slower. Nowadays BERT also has tiny/small versions available, such as the ones our company has open-sourced, which are roughly just as fast and perform better, unless you genuinely need the small model size.

So what does the xlarge version actually mean in practice? Some readers haven't even tried plain BERT because their machines can't run it; most readers with limited GPU memory have only run the base version of BERT and never managed to run, or never tried, the large version. xlarge is even bigger than large, and demands even more from your hardware. So frankly, for most readers there's simply no need to use ALBERT.

So why did the reputation of ALBERT being fast and good spread around? Aside from some irresponsible promotion by media accounts, I think this is largely due to the promotional efforts of the user brightmart.
It has to be said that brightmart made an indelible contribution to popularizing ALBERT in China. Even before the English ALBERT models were released, brightmart had already trained and open-sourced a Chinese version of ALBERT (albert_zh), and went on to train tiny, small, base, large, and xlarge versions in one go. At the time, BERT only had base and large versions, while ALBERT had tiny and small versions, and when people tried them out, they found them indeed much faster than BERT — leaving many with the impression that ALBERT is fast. In fact, ALBERT's speed has little to do with ALBERT itself; the key factor is the tiny/small size, and the corresponding tiny/small versions of BERT are just as fast...
Of course, you're welcome to think more deeply about the more fundamental reasons why parameter sharing works for ALBERT, or to study how to speed up inference after parameter sharing — these are both valuable questions worth investigating. I just wouldn't recommend using any ALBERT version below xlarge.

ELECTRA

ELECTRA comes from the paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Honestly, ELECTRA is a model that's hard to sum up in a few words — when it first appeared it excited a lot of people, but after its official open-source release it disappointed just as many. Its performance in practical applications today can't be called bad, but it can't really be called great either.

Characteristics

ELECTRA's starting point is the belief that BERT's approach of randomly masking a portion of tokens in the MLM objective is too simple, and that this difficulty should be increased. So it borrows ideas from GANs: it trains an MLM model (the generator) in the usual way, then uses this MLM model to sample replacements for tokens in the input sentence, and feeds the resulting sentence into another model (the discriminator), which judges which parts of the sentence have been replaced and which haven't. The generator and discriminator are trained simultaneously, so as the generator improves, the difficulty of the discrimination task gradually increases — the intuition being that this helps the model learn more valuable representations. In the end, only the discriminator's encoder is kept for use; the generator is typically discarded.

Because this progressive scheme makes the training process more targeted, ELECTRA's main highlight is greater training efficiency — according to the paper, it can reach the same performance as a BERT of the same size using only 1/4 of the training time, or even less. This is ELECTRA's main selling point.

Theory

However, in my view, ELECTRA is a model that doesn't quite hold up theoretically.

Why do I say this? ELECTRA's idea originates from GANs, but in computer vision, do we have examples of taking a trained GAN's discriminator and fine-tuning it for downstream tasks? At least I haven't seen any. In fact, this doesn't make sense theoretically. Take the original GAN: its optimal discriminator solution is $D(x)=\frac{p(x)}{p(x)+q(x)}$, where $p(x),q(x)$ are the distributions of the real and fake samples, respectively. Assuming training is stable and the generator has sufficiently strong fitting capacity, then as training proceeds, the fake samples will gradually converge toward the real samples, so $q(x)$ tends to $p(x)$, and $D(x)$ tends to the constant $1/2$. In other words, theoretically the final discriminator is just a constant function — how could you guarantee that it extracts good features?

Although ELECTRA is not exactly a GAN, the same issue applies: ELECTRA emphasizes that the MLM model serving as the generator must not be too powerful (otherwise, as noted above, the discriminator would degenerate into a constant). The paper states that the best results come when the generator's size is between $1/4$ and $1/2$ of the discriminator's size. This is where things start to get "mystical" — we've only just argued that too strong a generator doesn't work, but we haven't proven why a slightly weaker one does work, nor how much weaker is "just right," nor is it clear why training the generator and discriminator jointly should help. All of this now becomes pure "alchemy."

Performance

Of course, saying it's theoretically shaky doesn't mean I'm saying its performance is bad, and certainly not that the relevant benchmarks were faked. Because a constraint was placed on the generator's capacity, ELECTRA's training results are still meaningful, and its performance is decent enough — it's just that we went through a process of "the higher the expectations, the greater the disappointment."

ELECTRA's paper first appeared in the ICLR 2020 submissions, and the results at the time shocked everyone: roughly speaking, the small version of ELECTRA far outperformed the small version of BERT, even nearly matching the base version, while the base version of ELECTRA reached the level of large-version BERT. But when the code and weights were eventually released, the results shown on GitHub left people utterly stunned — performance had dropped by roughly 2 percentage points. Later, the authors clarified that the numbers in the paper were on the dev set, while those on GitHub were on the test set, which cleared things up a bit — but even so, ELECTRA's advantage over BERT in terms of performance essentially evaporated. (See the progression from ELECTRA: Surpassing BERT, the Best NLP Pretraining Model of 2019 to My Thoughts After the Release of ELECTRA's Source Code.)

Results reported in the ELECTRA paper

Results reported on ELECTRA's GitHub page

In fact, ELECTRA's evaluation on Chinese tasks more accurately reflects this. For example, in HIT's open-sourced Chinese-ELECTRA, ELECTRA performs about the same across tasks as similarly sized BERT models — with a slight edge on a few individual tasks, but nothing like the "crushing" results originally claimed.

Losing One While Gaining the Other

Some readers might think: even if performance is roughly the same, at least pretraining is faster — that's still an advantage worth acknowledging. But a new paper on arXiv from just the past couple of days suggests that ELECTRA's "roughly the same performance" might only be an illusion arising from simple tasks — if you build a slightly more complex task, ELECTRA gets thoroughly beaten by BERT.

This paper is titled Commonsense Knowledge Adversarial Dataset That Challenges ELECTRA. The authors constructed a new dataset called QADS based on SQuAD 2.0 using synonym substitution, and according to their tests, an ELECTRA large model that achieves 88% on SQuAD 2.0 only achieves 22% on QADS — and interestingly, BERT manages over 60%. Granted, this paper still looks fairly rough and hasn't yet received authoritative validation, so it shouldn't be taken at face value, but its results are already enough to make us reconsider ELECTRA. Previously, the paper Probing Neural Network Comprehension of Natural Language Arguments toppled BERT from its pedestal with just a single "not" — and it seems ELECTRA may suffer from this same kind of problem, possibly even more severely.

Setting aside other evidence, I personally feel that ELECTRA ultimately abandoning MLM is itself a case of "losing one thing while gaining another." If your starting premise is that MLM is too simple, you should just find a way to make MLM harder — why replace it with a discriminator altogether? It's entirely possible to improve the MLM model directly using a generator network (rather than replacing it with a discriminator). A recent Microsoft paper, Variance-reduced Language Pretraining via a Mask Proposal Network, offers exactly this kind of alternative approach: it lets the generator choose which positions to mask, rather than masking randomly. While I haven't reproduced its experiments, its overall reasoning feels quite convincing — unlike ELECTRA, which feels more like it was dreamed up out of thin air. Furthermore, I want to emphasize once more that MLM is genuinely useful — it's not merely a pretraining task, but also a model with practical value in its own right, as discussed in Do We Really Need GPT-3? No, BERT's MLM Model Can Also Do Few-Shot Learning.

Conclusion

So, after all that, the conclusion is: ELECTRA's pretraining speed is indeed faster, but based on current experiments, it shows no clear advantage over comparably sized BERT models on downstream tasks. You can give it a try, but there's no need to be too disappointed if the performance turns out worse. Also, if you need to use the MLM component's weights (for example, for UniLM-style text generation, see here), then ELECTRA won't work either, because ELECTRA's main body is the discriminator, not an MLM model — and the MLM model serving as ELECTRA's generator is simpler than the discriminator, and may suffer from insufficient fitting capacity or undertraining, making it a rather poor pretrained MLM model.

As for the underlying idea behind ELECTRA — improving on the overly simplistic random masking step of MLM — the direction seems correct so far, but whether replacing a generative model with a discriminative one is actually effective still needs further verification. Readers interested in digging deeper are welcome to think this through further and do more research.

Summary

This article has recorded my thoughts on ALBERT and ELECTRA, drawing mainly on my own experimental results as well as some references, in the hope of objectively laying out the strengths and weaknesses of these two models, so readers can feel more confident when choosing between them. Both models have their merits in specific scenarios, but both also come with certain limitations, and understanding these limitations and where they come from will help readers make better use of them.

I have no intention of unfairly maligning either model. If there's anything I've misunderstood, I welcome discussion in the comments.