T5 PEGASUS: Open-Sourcing a Chinese Generative Pretrained Model

T5 PEGASUS：开源一个中文生成式预训练模型 苏剑林 · 2021-03-03 · Machine Learning language model text generation attention

Last year, in the post That Chart-Topping T5 Model Can Now Be Played With in Chinese, we introduced Google's multilingual T5 model (mT5) and gave an example of using mT5 for Chinese text generation tasks. Admittedly, using mT5 for Chinese generation tasks is a workable solution, but a model that isn't trained entirely on Chinese corpora always feels a bit awkward, so we decided to build one ourselves.

After repeated deliberation and testing, we decided to use mT5 as the base architecture and initial weights, first refining the tokenizer to suit the characteristics of Chinese, and then following PEGASUS to construct a pretraining task, thereby training a new version of the T5 model. This is the T5 PEGASUS we're open-sourcing in this post.

Tokenizer

First, let's introduce our work on improving the tokenizer. The tokenizer used by mT5 is sentencepiece, a tokenization library written in C++ that is efficient and lightweight. Unfortunately, it isn't particularly friendly to Chinese, mainly for the following reasons:

1. sentencepiece forcibly converts certain full-width symbols to half-width symbols, which is unacceptable in some cases and may even affect task evaluation results;
2. Although the algorithm built into sentencepiece is capable of segmenting Chinese words, it's still not smart enough for Chinese word segmentation;
3. sentencepiece is written in C++, and although it's open source, for people used to Python, C++ is essentially a black box — the source code is hard to read and difficult to modify.

These characteristics led us to decide to switch the tokenizer back to BERT's tokenizer. However, simply replacing it with the original Chinese BERT tokenizer isn't enough: first, our previous work Faster Without Sacrificing Accuracy: WoBERT, a Word-Granularity Chinese BERT has already shown that using words as the basic unit produces better results for generative models; second, even looking purely at characters, the vocab.txt of Chinese BERT is quite incomplete, missing some common punctuation marks (such as double quotation marks) and Chinese characters (such as "琊"). For this reason, we chose to add word-segmentation capability to BERT's tokenizer and further improve vocab.txt.

Specifically, we added the top 200,000 words from Jieba word segmentation to the original Chinese BERT's token_dict, and then modified the tokenizer's logic so that it could split out words. These modifications have already been built into bert4keras, so you can just call them directly. Next, we used this modified tokenizer to traverse and segment our prepared pretraining corpus, counted the frequency of each token, and finally kept only the 50,000 most frequent tokens, yielding a vocab.txt of size 50,000 to build our final tokenizer.

Besides using this new tokenizer to train T5 PEGASUS, we also used it to retrain a new version of the WoBERT model (WoBERT+), which readers are welcome to try (link).

Pretraining Task

For the pretraining task, we wanted something closer to natural language generation (rather than just predicting masked-out spans as T5 does), while being as practically useful as possible. This led us to PEGASUS, from the paper PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. PEGASUS is described in its paper as a pretrained model specifically tailored for summarization, but in our view it can also serve as a general-purpose generative pretraining task. The general idea of PEGASUS is to construct summary-like data pairs using the longest common subsequence. T5 PEGASUS doesn't fully reproduce PEGASUS's approach — it merely borrows the idea for corpus construction.

Example of T5 PEGASUS training data

Specifically, suppose a document has $n$ sentences. We select about $n/4$ of them (which need not be contiguous), such that the text formed by concatenating these $n/4$ sentences and the text formed by concatenating the remaining $3n/4$ sentences have as long a common subsequence as possible. We then treat the text formed by the $3n/4$ sentences as the "source text" and the text formed by the $n/4$ sentences as the "summary." This gives us a pseudo-summary data pair of the form "(source text, summary)," which can then be used to train a Seq2Seq model. Note that if the document has no repeated sentences, then the source text and the summary won't share any sentences, so this generation task isn't simply copying the source text — it retains a certain degree of difficulty.

The search algorithm uses the following greedy method to progressively search until the length requirement is satisfied:

1. First find 1 sentence such that it has the longest common subsequence with the $n-1$ sentences generated so far;
2. Suppose $k$ sentences have already been found; then continue to find the $k+1$-th sentence, such that the text formed by concatenating these $k+1$ sentences has the longest common subsequence with the text formed by concatenating the remaining $n-k-1$ sentences.

Parameters and Configuration

The currently open-sourced T5 PEGASUS is the base version, with a total of 275 million parameters. During training, the maximum sequence length was 512, the batch size was 96, and the learning rate was $10^{-4}$. We trained it for 1 million steps using six 3090 GPUs, taking about 13 days. The data was a carefully processed general-purpose corpus of over 30 GB. Training accuracy reached about 47%, and training loss reached about 2.97. The model was written, trained, and tested using bert4keras.

GitHub repository: https://github.com/ZhuiyiTechnology/t5-pegasus

Experiments and Evaluation

On the two text generation tasks CSL and LCSTS, T5 PEGASUS is the SOTA among all models we're aware of:

$$\begin{array}{c} \text{CSL summarization results}\\ {\begin{array}{c|c|cccc} \hline & \text{beam size} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{BERT} & 1 & 63.81 & 65.45 & 54.91 & 45.52 \\ \text{WoBERT} & 1 & 66.38 & 68.22 & 57.83 & 47.76 \\ \text{mT5} & 1 & 66.96 & 69.00 & 58.74 & \textbf{49.79} \\ \text{T5 PEGASUS} & 1 & \textbf{67.68} & \textbf{69.87} & \textbf{59.8} & 49.37 \\ \hline \text{BERT} & 2 & 64.44 & 66.09 & 55.75 & 46.39 \\ \text{WoBERT} & 2 & 66.65 & 68.68 & 58.5 & 48.4 \\ \text{mT5} & 2 & 67.25 & 69.19 & 59.10 & \textbf{50.17} \\ \text{T5 PEGASUS} & 2 & \textbf{68.26} & \textbf{70.45} & \textbf{60.57} & 50.06 \\ \hline \text{BERT} & 3 & 64.75 & 66.34 & 56.06 & 46.7 \\ \text{WoBERT} & 3 & 66.83 & 68.81 & 58.67 & 48.6 \\ \text{mT5} & 3 & 67.17 & 69.11 & 59.05 & 50.13 \\ \text{T5 PEGASUS} & 3 & \textbf{68.39} & \textbf{70.54} & \textbf{60.69} & \textbf{50.19} \\ \hline \end{array}$$}\\

\\

\text{LCSTS Summary Generation Experiment Results}\\

{$$\begin{array}{c|c|cccc} \hline & \text{beam size} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{BERT} & 1 & 27.99 & 29.57 & 18.04 & 11.72 \\ \text{WoBERT} & 1 & \textbf{31.51} & 32.90 & 21.13 & 13.74 \\ \text{mT5} & 1 & 28.92 & 30.75 & 19.54 & 13.21 \\ \text{T5 PEGASUS} & 1 & 31.21 & \textbf{33.53} & \textbf{21.54} & \textbf{14.47} \\ \hline \text{BERT} & 2 & 29.20 & 30.70 & 19.17 & 12.64 \\ \text{WoBERT} & 2 & \textbf{31.91} & 33.35 & 21.55 & 14.13 \\ \text{mT5} & 2 & 29.96 & 31.67 & 20.40 & 13.84 \\ \text{T5 PEGASUS} & 2 & 31.47 & \textbf{34.00} & \textbf{21.98} & \textbf{14.75} \\ \hline \text{BERT} & 3 & 29.45 & 30.95 & 19.50 & 12.93 \\ \text{WoBERT} & 3 & \textbf{32.19} & 33.72 & 21.81 & 14.29 \\ \text{mT5} & 3 & 30.15 & 31.97 & 20.72 & 14.05 \\ \text{T5 PEGASUS} & 3 & 31.78 & \textbf{34.12} & \textbf{22.23} & \textbf{14.96} \\ \hline \end{array}$$}

\end{array}

More importantly, T5 PEGASUS demonstrates outstanding few-shot learning ability:

$$\begin{array}{c} \text{CSL summarization results (few-shot, beam size=1)}\\ {\begin{array}{c|c|cccc} \hline & \text{number of samples} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{WoBERT} & 10000 & 66.38 & 68.22 & 57.83 & 47.76 \\ \text{mT5} & 10000 & 66.96 & 69.00 & 58.74 & \textbf{49.79} \\ \text{T5 PEGASUS} & 10000 & \textbf{67.68} & \textbf{69.87} & \textbf{59.8} & 49.37 \\ \hline \text{WoBERT} & 1000 & 59.34 & 60.42 & 49.07 & 37.87 \\ \text{mT5} & 1000 & 59.91 & 61.52 & 50.38 & 40.87 \\ \text{T5 PEGASUS} & 1000 & \textbf{63.12} & \textbf{65.28} & \textbf{54.54} & \textbf{43.55} \\ \hline \text{WoBERT} & 100 & 55.68 & 55.33 & 43.10 & 31.55 \\ \text{mT5} & 100 & 55.33 & 54.62 & 42.78 & 32.50 \\ \text{T5 PEGASUS} & 100 & \textbf{60.87} & \textbf{62.78} & \textbf{52.30} & \textbf{41.40} \\ \hline \text{WoBERT} & 10 & 26.32 & 20.99 & 12.29 & 5.76 \\ \text{mT5} & 10 & 26.62 & 27.00 & 17.95 & 13.11 \\ \text{T5 PEGASUS} & 10 & \textbf{55.85} & \textbf{57.66} & \textbf{47.52} & \textbf{35.97} \\ \hline \end{array}$$}

\end{array}

Even when the number of labeled training samples is reduced to as few as 10, T5 PEGASUS can still be fine-tuned into a summary (title) generation model, with performance significantly exceeding other models. On LCSTS, T5 PEGASUS shows similar few-shot learning results, but since the non-T5-PEGASUS models perform far too poorly there, we haven't included that table here.

Few-Shot Demonstration

Below is a demonstration of the generation results from a model trained with only 10 labeled samples:

Input: Aiming at the problem of precisely measuring the reliability and fault tolerance of multiprocessor systems modeled on hypercube networks, and combining this with the characteristic that multiprocessor systems often experience structural failures when attacked by computer viruses, this paper studies the structural connectivity and substructure connectivity evaluation of n-dimensional hypercube networks. First, an upper bound on the 3-path structural connectivity is obtained by constructing a 3-path structural cut of the n-dimensional hypercube network; then, a lower bound on the 3-path substructure connectivity is obtained by constructing equivalence or reduction transformations of the 3-path substructure set of the n-dimensional hypercube network; finally, using the property that the 3-path structural connectivity of any network is no less than its 3-path substructure connectivity, it is confirmed that both the 3-path structural connectivity and substructure connectivity of the hypercube network equal the dimension of the hypercube network
Title: 3-Path Structural Connectivity and Substructure Connectivity of Hypercube Networks
Prediction: Research on Evaluation of Structural Connectivity and Substructure Connectivity of n-Dimensional Hypercube Networks
Input: Aiming at the problems of low prediction accuracy, large computational load, and high energy consumption in traditional wireless body area network (WBAN) prediction models for sensed data, this paper proposes an adaptive cubic exponential smoothing algorithm based on a penalty error matrix. First, a lightweight prediction model is established between sensing nodes and routing nodes; second, a carpet-search approach is used to optimize the parameters of the prediction model; finally, the penalty error matrix is used to further fine-tune the prediction model parameters. Experimental results show that, compared with the ZigBee protocol, within a range of 1000 time slots, the proposed method saves about 12% of energy; and compared with using the carpet-search approach alone, using the penalty error matrix improves prediction accuracy by 3.306%. The proposed method can further reduce WBAN energy consumption while effectively lowering computational complexity
Title: A Synchronous Prediction Energy-Saving Method for Wireless Body Area Networks Based on a Penalty Error Matrix
Prediction: Adaptive Cubic Exponential Smoothing Algorithm Based on a Penalty Error Matrix
Input: Aiming at problems such as low message transmission efficiency and large network resource overhead in the Internet of Vehicles (IoV) environment, this paper proposes a routing algorithm suitable for urban traffic scenarios based on cognitive interaction between vehicle nodes. First, based on trust theory, the concept of node cognitive interaction degree is proposed, and on this basis, vehicle nodes in the IoV are classified and assigned different initial values of cognitive interaction degree; at the same time, influencing factors such as vehicle node interaction time, interaction frequency, physical distance between vehicle nodes, hop count, and message lifetime are introduced, thereby constructing a vehicle node cognitive interaction evaluation model. Based on this model, the node's cognitive interaction degree is calculated and updated, and by comparing the cognitive interaction degree values between corresponding vehicle nodes, neighboring nodes with relatively high cognitive interaction degrees are selected as relay nodes for message forwar[ding]
Title: A Routing Algorithm Based on Node Cognitive Interaction in the Internet of Vehicles Environment
Prediction: A Routing Algorithm Based on Vehicle Node Cognitive Interaction
Input: Aiming at the joint estimation problem of near-field source direction of arrival (DOA) and distance, this paper proposes a near-field iterative adaptive algorithm (NF-IAA). First, a two-dimensional grid is used to represent all possible positions of sources in the near-field region, with each position regarded as a potential source incident on the array, thus representing the data model of the array output; then, through iterative cycling, the results of the previous spectral estimation are used to construct the signal's covariance matrix, and the inverse of the covariance matrix is used as a weighting matrix to estimate the potential source energy corresponding to each position; finally, a three-dimensional energy spectrum is plotted, and since only genuinely existing source energy is nonzero, the position corresponding to a spectral peak is the true position of an existing source. Simulation experiments show that under conditions of 10 snapshots, the DOA resolution probability of NF-IAA reaches 9[0%]
Title: Joint Estimation of Near-Field Source Two-Dimensional Parameters Based on an Iterative Adaptive Method
Prediction: Near-Field Iterative Adaptive Algorithm Based on NF-IAA
Input: Aiming at the problem that existing software crowdsourcing worker selection mechanisms give insufficient consideration to collaborative development among workers, this paper proposes a software crowdsourcing worker selection mechanism based on active-time grouping, building on the bidding model. First, crowdsourcing workers are divided into multiple collaborative development groups based on active time; then, the weight of each collaborative work group is calculated based on the development ability of workers within the group and a collaboration factor; finally, the collaborative work group with the highest weight is selected as the optimal work group, and the most suitable worker is selected from within that group for each task module based on module complexity. Experimental results show that, compared with the ability-priority selection method, this mechanism differs by only 0.57% in average worker ability, while reducing project risk by an average of 32% by ensuring collaboration among workers, effectively guiding crowdsourced software tasks that require multi-person collaborat[ion]
Title: A Software Crowdsourcing Worker Selection Mechanism Based on Active-Time Grouping
Prediction: A Software Crowdsourcing Worker Selection Mechanism Based on Active-Time Grouping

As we can see, even with very few labeled samples, the model is still able to produce quite readable generation results. This is thanks to the fact that PEGASUS-style pseudo-summary pretraining is very close to the downstream task.

A Brief Summary

This post mainly shared our Chinese generative pretrained model, T5 PEGASUS, which is built on mT5 and pretrained on Chinese corpora using PEGASUS-style pseudo-summary construction. It ultimately delivers solid text generation performance, especially outstanding few-shot learning ability. Readers with text generation needs are welcome to give it a try.