With bert4keras in hand, I've got my baseline covered: CLUE benchmark code

bert4keras在手，baseline我有：CLUE基准代码 苏剑林 · 2021-10-31 · Machine Learning model code Keras

CLUE (Chinese GLUE) is an evaluation benchmark for Chinese natural language processing, and it has by now gained recognition from quite a few teams. The official CLUE GitHub repo provides TensorFlow and PyTorch baselines, but they aren't easy to read, nor convenient to debug. In fact, whether it's TensorFlow or PyTorch, CLUE or GLUE, I've found that basically none of the baseline code out there could be called user-friendly — trying to understand it is a rather painful experience.

So I decided to implement a set of CLUE baselines based on bert4keras. After a period of testing, I've basically reproduced the officially claimed benchmark scores, and for some tasks even improved on them. Most importantly, all the code is kept as clear and readable as possible — truly "Deep Learning for Humans."

Code: https://github.com/bojone/CLUE-bert4keras

Code overview

Below I'll briefly introduce the design ideas behind the baselines for each task in this codebase. Before reading the article and the code, I'd suggest readers first take a look at the data format of each task on their own — I won't go into the details of the task data here. more

Text classification

First, IFLYTEK and TNEWS are two ordinary text classification tasks, so the approach is simple: just feed the standard "[CLS] + sentence + [SEP]" into BERT, and then take the [CLS] hidden vector for classification.

Diagram of the text classification model

Additionally, the pronoun disambiguation task WSC can also be converted into a single-text classification task. The original task is to judge whether two spans in a sentence (one of which is a pronoun) refer to the same entity; the baseline approach marks these two spans with different symbols in the text, and then directly feeds this marked-up text into BERT for binary classification.

Text matching

Next, AFQMC, CMNLI, and OCNLI are three text matching tasks. Text matching, put simply, is a sentence-pair classification task — for instance, similarity matching judges whether two sentences are similar, while natural language inference judges the logical relationship between two sentences (entailment, neutral, contradiction), etc. In the pretraining era, the standard approach for sentence matching tasks is to concatenate the two sentences with [SEP] and then treat it as a single-text classification task.

Diagram of the text matching model

It should be pointed out that in the original BERT, the SegmentID (called token type id in the original code) differs between the two sentences. But here, considering that models like RoBERTa have no NSP task and the SegmentID of 1 may never have been seen during pretraining, this implementation uses all-zero SegmentIDs. Experimental results show that this doesn't degrade text matching performance.

Similarly, in the CSL task, which judges whether an abstract description matches the given 4 keywords, we join the 4 keywords with a semicolon "；" and treat them as one sentence, which also converts it into a standard text matching problem.

Reading comprehension

Reading comprehension refers to the CMRC2018 task, which is an extractive reading comprehension task in the same format as SQUAD: a passage comes with multiple questions, each of which necessarily has an answer that is a span within the passage. The common approach is to concatenate the question and passage with [SEP] and feed them into BERT, then use two fully connected layers to predict the start and end positions separately. The problem with this is that it severs the connection between the start and end positions, and it also makes the behavior at training time and prediction time inconsistent.

The baseline here uses GlobalPointer as the output structure, which treats the start-end combination as a whole for classification. For details, see GlobalPointer: A Unified Approach to Nested and Non-Nested NER]. Using GlobalPointer makes the training and prediction behavior fully consistent, and noticeably speeds up decoding.

Diagram of the extractive reading comprehension model

Besides, whether it's SQUAD or CMRC2018, the passage length in most cases clearly exceeds 512, and some questions do have their answers located quite far into the passage — simply truncating the earlier part might cut off the answer entirely. If you use a model like NEZHA or RoFormer, you can directly process text longer than 512 tokens, but a model like BERT can't handle this well. To keep the code general-purpose, we retain the sliding-window design from the original BERT baseline here: the passage is split into multiple sub-passages with a stride of 128, and each sub-passage is combined with the question one at a time and fed into the model. After this splitting, it's possible that some sub-passages have "no answer" relative to the question — in that case, the answer is simply pointed to the [CLS] position$(0,0)$. At prediction time, long passages are split the same way, and the same question is answered against each sub-passage in turn, with the highest-scoring answer taken as the final result.

Multiple choice

"Multiple choice" here refers to the C3 task, which is also a kind of reading comprehension task — again, a passage comes with multiple questions, and the answer to each question is one of 4 given candidate answers, though it isn't necessarily a span found in the passage. The baseline approach for this kind of multiple-choice task might surprise a lot of people: it's essentially converted into a text matching problem, where each candidate answer is matched against the passage and question, and at prediction time the candidate with the highest score is chosen.

Diagram of the multiple-choice model

This means that a single original question now needs to be split into 4 samples to process, requiring 4 predictions to arrive at an answer, which greatly increases the computational cost. But, surprisingly, this approach basically outperforms all the other intuitively obvious baselines — it's much better than concatenating all the candidate answers together and doing 4-way classification. A similar task in English is DREAM], and the models on its leaderboard are basically variants of this same idea.

Idiom comprehension

The idiom reading comprehension task CHID is, in essence, also a multiple-choice reading comprehension problem, but its form is considerably more complex, so it's worth introducing separately.

Specifically, each CHID sample has 10 candidate idioms and several questions, each with several blanks, and the task is to decide which candidate idiom best fills each blank. If each question had only one blank, we could simply apply the multiple-choice approach from the previous section directly. But here each question may have multiple blanks, and the multiple-choice approach can only identify one blank at a time. So we use [unused0] to replace the blank we want to identify in the current pass, while any other, not-yet-identified blanks (if any) are replaced directly with 4 [MASK] tokens, for example:

[CLS] "This is actually an absurd farce. After Apple discovered that the owner of the iPad trademark on the mainland was not Taiwan Proview but rather Shenzhen Proview, it started to get anxious and [unused1]." Xiao Caiyuan said. In fact, two dramatic factors make the case even more [MASK] [MASK] [MASK] [MASK]. Materials submitted in the lawsuit Apple filed in a Hong Kong court show that IPADL is in fact a special-purpose company set up under the direction of Apple's lawyers, intended for acquiring the i-Pad trademark rights held by Proview. [SEP] settle the matter once and for all [SEP]

In other words, a question with multiple blanks gets split into several smaller sub-questions, and following the multiple-choice approach described above, each sub-question needs to be concatenated with the candidate answers for prediction — so the computational cost of each sub-question is equivalent to 10 ordinary classification samples. This is indeed rather expensive, but there's no way around it if we want good performance. To get a larger effective batch size, gradient accumulation is typically needed. Also, some questions are quite long and still need to be truncated; the truncation is done by centering on the blank currently being identified and extending as symmetrically as possible to the left and right.

Finally, according to the task design, each sample has several questions, and each blank across all questions shares the same 10 candidate idioms, but no answer is repeated across blanks. If, at prediction time, each blank simply independently takes the argmax answer, duplicate predictions may occur, which would violate the task design.

To ensure that predictions don't repeat, we need to use the "Hungarian algorithm": suppose there are $m$ blanks, each with $n > m$ candidate answers, giving us a $m\times n$ score matrix. We need to assign a different answer to each blank while maximizing the total score — mathematically, this is called the "assignment problem," and the standard solution is the "Hungarian algorithm," which we can solve directly using scipy.optimize.linear_sum_assignment. This kind of post-processing algorithm improves accuracy by about 6% compared to simply taking the argmax for each blank independently (which can lead to duplicate answers).

Entity recognition

The last task is CLUENER, a standard non-nested named entity recognition task. Common baselines are BERT+Softmax or BERT+CRF; here we use BERT+GlobalPointer instead — again, see GlobalPointer: A Unified Approach to Nested and Non-Nested NER]. When GlobalPointer is applied to NER, it can handle both nested and non-nested cases in a unified way. My repeated experiments show that, in the non-nested case, GlobalPointer can fully match the performance of CRF, while being faster at both training and prediction. So using GlobalPointer as the NER baseline is a natural choice.

Performance comparison

On the CLUE test set, the performance of each task is compared in the table below, where results marked $_{\text{-old}}$ are taken from the official CLUE results, and results marked $_{\text{-our}}$ are reproductions from this codebase. Here BERT and RoBERTa are both base-size: BERT is the original Chinese BERT released by Google, and RoBERTa is RoBERTa_wwm_ext, open-sourced by HIT. I'll test the large versions if and when I have the compute and the time.

$$\begin{array}{c} \text{classification task} \\ {\begin{array}{c|ccccccc} \hline & \text{IFLYTEK} & \text{TNEWS} & \text{AFQMC} & \text{CMNLI} & \text{OCNLI} & \text{WSC} & \text{CSL} \\ \hline \text{BERT}_{\text{-old}} & 60.29 & 57.42 & 73.70 & 79.69 & 72.20 & 74.60 & 80.36\\ \text{BERT}_{\text{-our}} & 61.19 & 56.29 & 73.37 & 79.37 & 71.73 & 73.85 & 84.03 \\ \hline \text{RoBERTa}_{\text{-old}} & 60.31 & \text{-} & 74.04 & 80.51 & \text{-} & \text{-} & 81.00\\ \text{RoBERTa}_{\text{-our}} & 61.12 & 58.35 & 73.61 & 80.81 & 74.27 & 82.28 & 85.33\\ \hline \end{array}} \end{array}$$

$$\begin{array}{c} \text{reading comprehension and NER tasks} \\ {\begin{array}{c|cccc} \hline & \text{CMRC2018} & \text{C3} & \text{CHID} & \text{CLUENER} \\ \hline \text{BERT}_{\text{-old}} & 71.60 & 64.50 & 82.04 & 78.82\\ \text{BERT}_{\text{-our}} & 72.10 & 61.33 & 85.13 & 78.68\\ \hline \text{RoBERTa}_{\text{-old}} & 75.20 & 66.50 & 83.62 & \text{-}\\ \text{RoBERTa}_{\text{-our}} & 75.40 & 67.11 & 86.04 & 79.38\\ \hline \end{array}} \end{array}$$

Note: TNEWS and WSC are left blank here because their test sets were updated later, but the official GitHub repo did not update the corresponding RoBERTa results in time; OCNLI and CLUENER are left blank because the official repo only tested BERT base and RoBERTa large — no RoBERTa base results were given.

Summary

This post shared the CLUE evaluation benchmark code I built based on bert4keras, along with a brief introduction to the modeling approach for each category of task. This set of baseline code is simple, clear, and easy to adapt, and it basically achieves the benchmark scores officially claimed by CLUE — for some tasks, even better. So I'd say it counts as a passing-grade baseline codebase.