NSP, a Once-Disdained Pretraining Task, Delivers Impressive Zero-Shot Results
Among the many pretraining task designs out there, NSP has generally been regarded as one of the worse choices, since it's relatively easy and including it in pretraining doesn't bring any clear benefit to downstream fine-tuning—in fact, the RoBERTa paper even showed it can have a negative effect. As a result, subsequent pretraining work has typically gone one of two ways: either drop NSP entirely, as RoBERTa did, or find ways to make NSP harder, as ALBERT did. In short, NSP has long been the "unloved child" of pretraining tasks.
But now there's a plot twist—NSP may finally get its comeback. A recent paper, NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task—Next Sentence Prediction (hereafter NSP-BERT), shows that NSP can actually achieve remarkably good zero-shot performance! This is yet another classic case of prompt-based few/zero-shot learning—except this time, the star of the show is NSP.
Background
We used to think of pretraining as purely pretraining—a way to provide better initialization for downstream task training. BERT's pretraining tasks, MLM (Masked Language Model) and NSP (Next Sentence Prediction), went largely unexamined in their own right for quite a long time; people focused almost entirely on how to fine-tune to get better downstream performance. Even T5, with its 11 billion parameters, still followed the "pretrain + fine-tune" paradigm. more
The first strong challenge to this mindset came from last year's release of GPT-3, which showed that with a sufficiently large pretrained model, you could design specific prompts to achieve excellent few/zero-shot performance without any fine-tuning at all. Wherever GPT goes, BERT is never far behind—if GPT can do it, surely BERT can too. This led to the later PET work, which likewise constructs special prompts and leverages a pretrained MLM model for few/zero-shot learning. Readers unfamiliar with this can refer to Do We Really Need GPT-3? No, BERT's MLM Can Also Do Few-Shot Learning.
Since then, "pretraining + prompt" work has kept growing, and now it's practically exploding. This whole line of work is collectively known as "prompt-based language models," and a quick search will turn up plenty of examples. By now, a consensus has emerged: constructing an appropriate prompt so that the downstream task's format more closely resembles the pretraining task tends to yield better results. So how to construct prompts has become one of the core concerns of this line of research—P-tuning, for instance, is a classic example (see P-tuning: Automatically Constructing Templates to Unleash the Potential of Language Models).
NSP Enters the Scene
Take a closer look at prompt-based work, and you'll notice that most current research focuses on how to better exploit pretrained GPT, MLM, or encoder-decoder models, while little attention has been paid to other pretraining tasks. The NSP-BERT paper, however, fully taps into the potential of the NSP task, and it inspires us to see that even within the confines of prompt-based methods, there's still plenty of room for divergent research directions.
The so-called NSP task isn't really about predicting the next sentence—it's about judging, given two sentences, whether they are adjacent. Correspondingly, the idea behind NSP-BERT is actually quite simple: taking classification as an example, treat the input as the first sentence, then append a specific prompt to each candidate category to form the second sentence, and judge, one by one, which second sentence is most coherent with the first. You'll notice this idea is very similar to PET's—indeed, most prompt-based work is easy to understand once explained; the hard part is coming up with the idea in the first place.
The figure below illustrates some reference prompt schemes NSP-BERT uses for common NLU tasks, showing that it can handle quite a few tasks:
Some prompts used by NSP-BERT for common NLU tasks
Looking at this figure alone gives you the gist of most of NSP-BERT's ideas; the rest of the paper mainly elaborates on the details shown here. Readers wanting a deeper dive should go read the original paper carefully themselves.
This kind of setup for NSP-BERT isn't actually the first of its kind—earlier work proposed using NLI models for zero-shot classification (see NLI Models as Zero-Shot Classifiers), whose format is essentially the same as NSP's. But that approach required supervised fine-tuning on labeled corpora, whereas using purely unsupervised NSP this way is a first.
Experimental Results
Interestingly, for us, NSP-BERT is a refreshingly down-to-earth piece of work. For instance, it's written by Chinese researchers, its experimental tasks are all in Chinese (FewCLUE and DuEL2.0), and the code is open-sourced. Here's the author's repository:
Github: https://github.com/sunyilgdx/NSP-BERT
Most importantly, NSP-BERT's results are genuinely impressive:
NSP-BERT's zero-shot results
Results on the entity linking task
Effect of model scale on performance
Overall, after reading through these results, all I can say to NSP is: "My apologies, I underestimated you." Such a heavyweight in the world of models has been standing right in front of us all along, and we simply hadn't noticed—full credit has to go to the NSP-BERT authors for their keen insight.
Summary
This post introduced a paper that uses BERT's NSP pretraining task for zero-shot learning. The results show that NSP can achieve remarkably good zero-shot performance—perhaps, given time, NSP is about to rise again.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.