SimBERTv2 is here! RoFormer-Sim, a model that combines retrieval and generation
Last year we released the SimBERT model, which turned out to be one of our more successful open-source releases, earning quite a bit of recognition from readers. In short, SimBERT is a model that integrates both generation and retrieval: it can serve as a fairly strong baseline for sentence embeddings, and it can also be used to automatically generate similar questions, functioning as an auxiliary data augmentation tool — this latter capability was pioneering at the time.
Recently, building on RoFormer as the base architecture, we have further integrated and optimized the technology behind SimBERT, and finally released the upgraded RoFormer-Sim model.
Introduction
RoFormer-Sim is the upgraded version of SimBERT — we might informally call it "SimBERTv2," with SimBERT now referring to the old version by default. From the outside, apart from switching the base architecture to RoFormer, RoFormer-Sim doesn't look much different from SimBERT. In fact, the main differences lie in the training details, which we can compare using two formulas:
$$\begin{array}{c} \text{SimBERT} = \text{BERT} + \text{UniLM} + \text{contrastive learning} \\[5pt] \text{RoFormer-Sim} = \text{RoFormer} + \text{UniLM} + \text{contrastive learning} + \text{BART} + \text{distillation}\\ \end{array}$$more
Beyond that, RoFormer-Sim uses more training data and has been extended to general sentence types. That is, unlike SimBERT, which was limited to interrogative sentences, RoFormer-Sim can be used to generate similar sentences for general sentence types as well, giving it a much broader range of applications. Other training details include RoFormer-Sim using a larger batch size and maxlen, among other things, which we'll introduce further below.
Open-source repository: https://github.com/ZhuiyiTechnology/roformer-sim
Corpus
The key to both SimBERT and RoFormer-Sim lies in the construction of the training corpus. The training corpus for RoFormer-Sim consists of two parts: 1) similar-sentence pairs of the interrogative type; 2) similar-sentence pairs of the general type. For interrogative-type similar sentences, we followed the same approach as SimBERT — collecting similar questions from Baidu Zhidao and then further cleaning them with rules; this part of the pipeline is already quite mature for us. For general-type similar sentences, we had no ready-made source to collect from, so we proposed two schemes that allow us to construct (pseudo) similar sentence pairs in a largely unsupervised way.
The first scheme is based on the idea that "answers to the same question are similar." Given an existing QA corpus where the same question has multiple answers, we can split each answer into sentences, then use an off-the-shelf similarity function to compare the sentences across answers, and pick out sentence pairs whose similarity exceeds some threshold to use as similar-sentence pairs.
The second scheme is based on the idea that "sentences within the same passage are similar." It's even simpler and more direct: split each passage into sentences, compute pairwise similarity using an off-the-shelf similarity function, and pick out sentence pairs whose similarity exceeds some threshold as similar-sentence pairs. Clearly this scheme is less well-justified, so we set its threshold higher.
This involves an "off-the-shelf similarity function," for which we directly use a variant of Jaccard similarity — in other words, all we need is a rule-based, character-level similarity measure. The semantic relatedness is instead captured through the internal coherence of the passage and the generalization ability of the pretrained model itself. Using the first scheme, we constructed about 4.5 million (pseudo) similar sentence pairs from several reading comprehension datasets; using the second scheme, we constructed about 4.7 million (pseudo) similar sentence pairs from over 30GB of parallel corpora; meanwhile, the crawled questions amounted to roughly 30 million groups of similar questions (each group can form multiple pairs). From this perspective, the number of questions far exceeds that of general sentences, so we sampled from them at a 1:1 ratio to keep the amount of data for each sentence type balanced.
Generation
The training procedure for RoFormer-Sim is essentially the same as for SimBERT, as shown in the figure below. The one difference is that, to strengthen the model's generative ability, when constructing the training corpus we also randomly replace some tokens of the input sentence with [MASK]. This pretraining approach was first proposed by BART. The difference between our approach and BART's is: BART is "input a noisy sentence, output the original sentence," whereas ours is "input a noisy sentence, output a sentence similar to the original" — in theory, our task is even harder.
Diagram of SimBERT's training scheme
There's no particularly good quantitative metric for evaluating generation quality, so let's just eyeball a few examples directly:
gen_synonyms(u'广州和深圳哪个好？')
[
'深圳和广州哪个好？',
'广州和深圳哪个好',
'广州和深圳哪个更好？',
'深圳和广州哪个更好？',
'深圳和广州，那个更好？',
'深圳和广州哪个好一些呢？',
'深圳好还是广州好？',
'广州和深圳哪个地方好点？',
'广州好还是深圳好？',
'广州和深圳哪个好一点',
'广州和深圳哪个发展好？',
'深圳好还是广州好',
'深圳和广州哪个城市更好些',
'深圳比广州好吗？',
'到底深圳和广州哪个好？为什么呢？',
'深圳究竟好还是广州好',
'一般是深圳好还是广州好',
'广州和深圳那个发展好点',
'好一点的深圳和广州那边好？',
'深圳比广州好在哪里？'
]
gen_synonyms(u'科学技术是第一生产力。')
[
'科学技术是第一生产力！',
'科学技术是第一生产力',
'一、科学技术是第一生产力。',
'一是科学技术是第一生产力。',
'第一，科学技术是第一生产力。',
'第一生产力是科学技术。',
'因为科学技术是第一生产力。',
'科学技术是第一生产力知。',
'也即科学技术是第一生产力。',
'科学技术是第一生产力吗',
'科技是第一生产力。',
'因此，科学技术是第一生产力。',
'其次，科学技术是第一生产力。',
'科学技术才是第一生产力。',
'科学技术是第一生产力吗？',
'第二，科学技术是第一生产力。',
'所以说科学技术是第一生产力。',
'科学技术确实是第一生产力。',
'科学技术还是第一生产力',
'科学技术是第一生产力对吗？'
]
Overall, we've achieved preliminary similarity augmentation for arbitrary sentence types, though the augmentation quality for questions is better than for general sentence types — this is because the quality of the question data in the training corpus is noticeably higher than that of the general sentence data. Since we adopted BART-style training, besides generating similar sentences directly, we can also manually mask out certain parts and let the model freely expand on them, for example:
gen_synonyms(u'科学技术是第一生产力。', mask_idxs=[6, 7]) # mask掉“第一”
[
"科学技术是第一生产力",
"2、科学技术是第一生产力。",
"科学技术是第一生产力，也是第二生产力。",
"科学技术是第一生产力，科学发展是第二生产力。",
"9、科学技术是第一生产力。",
"第一，科学技术是一种生产力。",
"科学技术是生产力。",
"科学技术是第二生产力。",
"科学技术是第一生产力”现在提出来的。",
"一、科学技术是一种生产力。",
"科学技术是第一生产力是什么意思",
"科学技术是一种主要生产力。",
"一：科学技术是最高生产力。",
"指科学技术不是第一生产力。",
"科学技术是第二生产力，第一生产力又是第二生产力。",
"二、科学技术是一种生产力。",
"世界上第一种生产力是科学技术。",
"科学技术是社会主义生产力之一。",
"第二，科学技术也是第二生产力。",
"科技是一切生产力。"
]
We'll leave further exploration of what else you can do with this to readers.
Retrieval
Adding general-sentence-type corpora and introducing BART-style training both improved the generation model's performance to some extent. Unexpectedly, however, we found that the retrieval model (i.e., the sentence encoding model) actually got worse. Our best guess is that although more corpora and larger noise make the generation task harder, for contrastive learning these differently-typed or noisy samples, when used as negative samples, actually become easier to distinguish. For instance, if a batch contains both interrogative and declarative sentences, the model can simply identify many negatives based on sentence type (rather than semantics), which weakens its ability to understand semantics.
Of course, both SimBERT and RoFormer-Sim are fundamentally positioned as similar-sentence augmentation models — the retrieval model is merely a "by-product." Still, we'd like this "by-product" to be as good as possible. To that end, after training RoFormer-Sim, we further transferred SimBERT's retrieval performance onto RoFormer-Sim through distillation, so that RoFormer-Sim's retrieval performance ends up roughly on par with, or even better than, SimBERT's. The distillation method is simple: suppose for the same batch of sentences, the sentence vectors produced by SimBERT are $u_1, u_2, \cdots, u_n$, and those produced by RoFormer-Sim are $v_1, v_2, \cdots,v_n$. Then we train using
\begin{equation}\mathcal{L}_{\text{sim}} = \frac{\lambda}{n^2}\sum_{i=1}^n\sum_{j=1}^n (\cos(u_i,u_j)-\cos(v_i,v_j))^2\end{equation}
as the loss, where $\lambda=100$. Of course, to prevent the model from "forgetting" the generation task, we also add the generation loss during distillation, i.e., $\mathcal{L}=\mathcal{L}_{\text{sim}}+\mathcal{L}_{\text{gen}}$. For the base version, distillation doesn't require many steps — roughly 5,000 steps is enough to complete training.
As in Which Unsupervised Semantic Similarity Method Wins? We Ran a Fairly Comprehensive Benchmark, we compared the retrieval performance of SimBERT and RoFormer-Sim using the same set of tasks (where the three numbers in each cell correspond respectively to "without whitening," "with whitening," and "with whitening-256," as in that earlier benchmark):
$$\small{\begin{array}{l|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{STS-B} \\ \hline \text{V1}\text{-P1} & 38.50 / \color{red}{23.64} / \color{red}{30.79} & 48.54 / \color{red}{31.78} / \color{red}{40.01} & 76.23 / \color{red}{75.05} / \color{red}{74.50} & 15.10 / \color{green}{18.49} / \color{green}{15.64} & 74.14 / \color{red}{73.37} / \color{green}{75.29} \\ \text{V1}\text{-P2} & 38.93 / \color{red}{27.06} / \color{red}{30.79} & 49.93 / \color{red}{35.38} / \color{red}{40.14} & 75.56 / \color{red}{73.45} / \color{red}{74.39} & 14.52 / \color{green}{18.51} / \color{green}{15.74} & 73.18 / \color{green}{73.43} / \color{green}{75.12} \\ \text{V1}\text{-P3} & 36.50 / \color{red}{31.32} / \color{red}{31.24} & 45.78 / \color{red}{29.17} / \color{red}{40.98} & 74.42 / \color{red}{73.79} / \color{red}{73.43} & 15.33 / \color{green}{18.39} / \color{green}{15.87} & 67.31 / \color{green}{70.70} / \color{green}{72.00} \\ \text{V1}\text{-P4} & 33.53 / \color{red}{29.04} / \color{red}{28.78} & 45.28 / \color{red}{34.70} / \color{red}{39.00} & 73.20 / \color{red}{71.22} / \color{red}{72.09} & 14.16 / \color{green}{17.32} / \color{green}{14.39} & 66.98 / \color{green}{70.55} / \color{green}{71.43} \\ \hline \text{V2}\text{-P1} & 39.52 / \color{red}{25.31} / \color{red}{31.10} & 50.26 / \color{red}{33.47} / \color{red}{40.16} & 76.02 / \color{red}{74.92} / \color{red}{74.58} & 14.37 / \color{green}{19.31} / \color{green}{14.81} & 74.46 / \color{red}{71.00} / \color{green}{76.29} \\ \text{V2}\text{-P2} & 39.71 / \color{red}{32.60} / \color{red}{30.89} & 50.80 / \color{red}{37.62} / \color{red}{40.12} & 75.83 / \color{red}{73.45} / \color{red}{74.52} & 13.87 / \color{green}{19.50} / \color{green}{14.88} & 73.47 / \color{green}{74.56} / \color{green}{76.40} \\ \text{V2}\text{-P3} & 39.55 / \color{red}{24.61} / \color{red}{31.82} & 50.25 / \color{red}{29.59} / \color{red}{41.43} & 74.90 / \color{red}{73.95} / \color{red}{74.06} & 14.57 / \color{green}{18.85} / \color{green}{15.26} & 68.89 / \color{green}{71.40} / \color{green}{73.36} \\ \text{V2}\text{-P4} & 36.02 / \color{red}{29.71} / \color{red}{29.61} & 48.22 / \color{red}{35.02} / \color{red}{39.52} & 73.76 / \color{red}{71.19} / \color{red}{72.68} & 13.60 / \color{green}{16.67} / \color{green}{13.86} & 68.39 / \color{green}{71.04} / \color{green}{72.43} \\ \hline \end{array}}$$
As the table shows, whether or not whitening is applied, RoFormer-Sim outperforms SimBERT on most tasks. This confirms that the distilled RoFormer-Sim does indeed achieve better retrieval performance, so this "by-product" turns out not to be too bad after all.
Using the same method, we also produced a small version of RoFormer-Sim, this time using the base version of RoFormer-Sim as the teacher model for distillation. However, this required substantially more distillation steps (around 500,000). The final results are as follows:
$$\small{\begin{array}{l|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{STS-B} \\ \hline \text{V1}_{\text{small}}\text{-P1} & 30.68 / \color{red}{27.56} / \color{red}{29.07} & 43.41 / \color{red}{30.89} / \color{red}{39.78} & 74.73 / \color{red}{73.21} / \color{red}{73.50} & 15.89 / \color{green}{17.96} / \color{green}{16.75} & 70.54 / \color{green}{71.39} / \color{green}{72.14} \\ \text{V1}_{\text{small}}\text{-P2} & 31.00 / \color{red}{29.14} / \color{red}{29.11} & 43.76 / \color{red}{36.86} / \color{red}{39.84} & 74.21 / \color{red}{73.14} / \color{red}{73.67} & 16.17 / \color{green}{18.12} / \color{green}{16.81} & 70.10 / \color{green}{71.40} / \color{green}{72.28} \\ \text{V1}_{\text{small}}\text{-P3} & 30.03 / \color{red}{21.24} / \color{red}{29.30} & 43.72 / \color{red}{31.69} / \color{red}{40.81} & 72.12 / \color{red}{70.27} / \color{red}{70.52} & 16.93 / \color{green}{21.68} / \color{green}{18.75} & 66.55 / \color{red}{66.11} / \color{green}{69.19} \\ \text{V1}_{\text{small}}\text{-P4} & 29.52 / \color{red}{28.41} / \color{red}{28.57} & 43.52 / \color{red}{36.56} / \color{red}{40.49} & 70.33 / \color{red}{68.75} / \color{red}{69.01} & 15.39 / \color{green}{21.57} / \color{green}{16.34} & 64.73 / \color{green}{68.12} / \color{green}{68.24} \\ \hline \text{V2}_{\text{small}}\text{-P1} & 37.33 / \color{red}{23.59} / \color{red}{31.31} & 47.90 / \color{red}{29.21} / \color{red}{42.07} & 74.72 / \color{green}{74.94} / \color{red}{74.69} & 13.41 / \color{green}{15.30} / \color{green}{13.61} & 71.48 / \color{red}{69.01} / \color{green}{75.10} \\ \text{V2}_{\text{small}}\text{-P2} & 37.42 / \color{red}{31.25} / \color{red}{31.18} & 49.15 / \color{red}{38.01} / \color{red}{41.98} & 75.21 / \color{red}{73.47} / \color{red}{74.78} & 13.38 / \color{green}{15.87} / \color{green}{13.69} & 72.06 / \color{green}{73.92} / \color{green}{75.69} \\ \text{V2}_{\text{small}}\text{-P3} & 36.71 / \color{red}{30.33} / \color{red}{31.25} & 49.73 / \color{red}{31.03} / \color{red}{42.74} & 74.25 / \color{red}{72.72} / \color{red}{74.19} & 14.58 / \color{green}{18.68} / \color{red}{14.40} & 69.12 / \color{green}{71.07} / \color{green}{72.68} \\ \text{V2}_{\text{small}}\text{-P4} & 32.80 / \color{red}{27.87} / \color{red}{29.65} & 46.80 / \color{red}{36.93} / \color{red}{41.31} & 72.30 / \color{red}{69.94} / \color{green}{72.38} & 13.45 / \color{green}{16.93} / \color{red}{13.38} & 67.21 / \color{green}{70.42} / \color{green}{71.39} \\ \hline \end{array}}$$
Summary
This post introduced and released our upgraded version of SimBERT — RoFormer-Sim (SimBERTv2), which can be used both for augmenting similar sentences and as a fairly strong baseline for semantic similarity tasks. Compared with SimBERT, its biggest feature is that it extends coverage to general sentence types, no longer being limited to similar questions. We look forward to readers exploring and sharing more creative uses for it~
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.