Short-Text Matching Baseline: Attempting Pretrained Models on Anonymized Data

短文本匹配Baseline：脱敏数据使用预训练模型的尝试 苏剑林 · 2021-03-05 · Machine Learning language model semantics semantic similarity

Recently I joined in the fun by trying out the "Xiaobu Assistant Dialogue Short-Text Semantic Matching" track of the Global AI Technology Innovation Competition. The task itself is the usual short-text sentence-pair binary classification problem, which is no longer anything special in this era when pretrained Transformers "run rampant" everywhere. What makes this competition interesting, though, is that the data had been anonymized (desensitized) — every character was mapped to a numeric ID, so we can't access the original text.

Under these circumstances, can we still use pretrained models like BERT? Yes, we can, but it takes some tricks, and it may also require an additional round of pretraining. In this post I'll share a baseline that combines classification, pretraining, and semi-supervised learning all together, which can be applied to tasks with anonymized data.

Model Overview

The idea behind the whole model is essentially a variant of PET (Pattern-Exploiting Training), introduced in the earlier post Do We Really Need GPT-3? No, BERT's MLM Can Do Few-Shot Learning Too, where a single MLM model handles everything. Here's a schematic:

Schematic of the model in this post

As you can see, the entire model is just an MLM model. Specifically, we add two special tokens, [YES] and [NO], to the vocabulary to represent the similarity between sentences, and we use the output vector corresponding to [CLS] to predict the sentence-pair label ([YES] or [NO]). To build the training corpus, we simply concatenate the sentence pairs in the usual way, randomly mask some tokens in the two sentences, and then predict those masked tokens at the corresponding output positions.

In this way, we simultaneously perform the sentence-pair classification task (the prediction at [CLS]) and the MLM pretraining task (predicting the other masked tokens). Moreover, samples without labels (such as the test set) can also be thrown into training, as long as we don't try to predict [CLS] for them. So through a single MLM model, we manage to combine classification, pretraining, and semi-supervised learning all at once.

Reusing BERT

Can BERT still be used with anonymized data? Of course. For BERT, anonymized data really just means a different embedding layer — the rest of the layers are still valuable. So reusing BERT mainly comes down to re-aligning the embedding layer through pretraining.

In this process, initialization matters a lot. First, we take out the special tokens like [UNK], [CLS], [SEP] from BERT's embedding layer and leave them unchanged. Then, we separately count character frequencies in the anonymized (ciphertext) data and in plaintext data — the plaintext data can be any general-purpose open-source corpus, not necessarily the actual plaintext corresponding to the ciphertext data. Next, we simply align the plaintext vocabulary with the ciphertext vocabulary according to frequency ranking. This lets us pull out BERT's embedding vectors according to the plaintext characters and use them as the corresponding initialization.

In short, I use BERT's embedding for the highest-frequency plaintext character to initialize the highest-frequency ciphertext character, and so on down the list, to perform a basic vocabulary alignment. My own comparative experiments show that this trick noticeably speeds up model convergence.

Code

That about covers the model. With this approach, using the base version of BERT, I currently get a leaderboard score of 0.866, while offline it's already at 0.952 (single model, no K-fold ensembling — it seems everyone's online/offline gap is pretty large). Here I'm sharing my bert4keras implementation:

GitHub repo: https://github.com/bojone/oppo-text-match

I've already computed and included the plaintext character frequency statistics, also synced to GitHub, so feel free to just use them directly. I'd suggest training for 100 epochs, which takes about 6 hours on a 3090.

By the way, if you want to use the Large version of BERT, I don't recommend the RoBERTa-wwm-ext-large released by HIT (Harbin Institute of Technology) — the reason was already explained in Do We Really Need GPT-3? No, BERT's MLM Can Do Few-Shot Learning Too: for some reason that version has randomly initialized MLM weights, and we specifically need the MLM weights. If you need a Large version, I recommend Tencent UER's open-source BERT Large.

Summary

Nothing too fancy here — just sharing a simple baseline for the competition, and taking the opportunity to write up a blog post. Hope it helps!