Now You Can Play with Chinese GPT2 in Keras (GPT2_ML)

现在可以用Keras玩中文GPT2了（GPT2_ML） 苏剑林 · 2020-03-16 · Machine Learning language model NLP text generation attention

A while back I noticed that some heavyweight had open-sourced a Chinese GPT2 model — the biggest version, with 1.5 billion parameters. Looking at the demo the author provided, the generation quality was genuinely impressive, so I thought I'd load it into my own bert4keras and give it a try. The trouble was, the early version of bert4keras had a fairly rigid overall architecture, which made it inconvenient to integrate multiple different models. A couple of weeks ago I finally couldn't stand it anymore, and rewrote the whole structure of bert4keras. Now bert4keras can flexibly build all kinds of Transformer-based model architectures — for instance, GPT2 and T5 are both already integrated in it.

A Quick Primer on GPT2

GPT — I'm sure many readers have heard of it. In short, it's a language model based on the Transformer architecture, introduced in the paper GPT: Improving Language Understanding by Generative Pre-Training. But it wasn't born just to be a language model — rather, it uses a language model as a way to pretrain itself, and then fine-tunes on downstream tasks to improve their performance. It's the pioneer of the "Transformer + pretraining + fine-tuning" paradigm; by comparison, even BERT counts as its "junior." GPT2, meanwhile, is the upgraded version of GPT — a bigger model trained on more data, with the largest variant reaching 1.5 billion parameters. more

Chinese Version

Most readers who've seen popular-science posts about GPT2 come away impressed by its generation quality. But no matter how good it is, it's still someone else's language — OpenAI never bothered training a Chinese version. The good news, though, is that a project called GPT2_ML has open-sourced a Chinese version of GPT2, and it too is the largest 1.5-billion-parameter model.

The GPT2 currently integrated into bert4keras is precisely the one from the GPT2_ML project, rather than OpenAI's original, since bert4keras prioritizes serving the Chinese-language community. It's worth pointing out that the model architecture of GPT2_ML differs both from OpenAI's GPT2 and from BERT. Here's a comparison of the Block structures of all three:

Block diagram of the official GPT2Block diagram of BERTBlock diagram of GPT2_ML

Let's Test It Out

First, download the model weights, available here:

Link: https://pan.baidu.com/s/1OXBd16o82SpIzu57kwA8Mg Extraction code: q79r
The main file, "model.ckpt-100000.data-00000-of-00001," can also be downloaded from Google Drive. After downloading, please check the SHA256 checksum of model.ckpt-100000.data-00000-of-00001 (4a6e5124df8db7ac2bdd902e6191b807a6983a7f5d09fb10ce011f9a073b183e).

Then install bert4keras version 0.6.0 or above (the latest version at time of writing), and you'll be able to run the test code below (if it has become outdated due to subsequent version changes, please check the latest version at basic_language_model_gpt2_ml.py):

#! -*- coding: utf-8 -*-
# 基本测试：中文GPT2模型
# 介绍链接：https://kexue.fm/archives/7292

import numpy as np
from bert4keras.models import build_transformer_model
from bert4keras.tokenizers import Tokenizer
from bert4keras.snippets import AutoRegressiveDecoder
from bert4keras.snippets import uniout

config_path = '/root/gpt2/config.json'
checkpoint_path = '/root/gpt2/model.ckpt-100000'
dict_path = '/root/gpt2/vocab.txt'

tokenizer = Tokenizer(dict_path,
                      token_start=None,
                      token_end=None,
                      do_lower_case=True)  # 建立分词器

model = build_transformer_model(config_path=config_path,
                                checkpoint_path=checkpoint_path,
                                model='gpt2_ml')  # 建立模型，加载权重

class ArticleCompletion(AutoRegressiveDecoder):
    """基于随机采样的文章续写
    """
    @AutoRegressiveDecoder.set_rtype('probas')
    def predict(self, inputs, output_ids, step):
        token_ids = np.concatenate([inputs[0], output_ids], 1)
        return model.predict(token_ids)[:, -1]

    def generate(self, text, n=1, topk=5):
        token_ids, _ = tokenizer.encode(text)
        results = self.random_sample([token_ids], n, topk)  # 基于随机采样
        return [text + tokenizer.decode(ids) for ids in results]

article_completion = ArticleCompletion(start_id=None,
                                       end_id=511,  # 511是中文句号
                                       maxlen=256,
                                       minlen=128)

print(article_completion.generate(u'今天天气不错'))

Some sample outputs:

article_completion.generate(u'今天天气不错')
[u'The weather is nice today, so I could go for a run. Last night I watched a documentary about running, in which the female lead talks about a girl's growth story — very inspiring, and very beautiful. I want to go running too, but I didn't know I needed running shoes, so I bought a pair. The documentary was actually about running shoes; there was one episode about a girl who never wore running shoes from elementary school onward, and only started trying running in high school.']
article_completion.generate(u'双十一')
[u'Double Eleven is coming soon! Are you worried about logistics and delivery for Double Eleven? Worried you won't have time to pick up your packages from the warehouse? Worried about not knowing how to find good deals? Frustrated that you can't get the product you want? Well, Double Eleven is here! Today, let's take a look at how these couriers deliver packages! 1. Logistics and delivery: courier companies are responsible for dispatching packages and providing logistics services.']
article_completion.generate(u'科学空间')
[u'The Science Space Station (English: science space station) is a space station of the People's Republic of China. The station was developed by the Dalian Institute of Physics, Chinese Academy of Sciences, and mainly uses nuclear-powered propulsion systems developed by the same institute. The Science Space Station is located in Haidian District, Beijing, 393 meters above the ground, with a total floor area of about 10,000 square meters and a total investment of roughly 500 million RMB. Construction on the Science Space Station began on December 26, 2018, and it was completed and put into operation in June 2021.']

Pretty impressive, right?

Want to Fine-tune It?

Seeing results like these, I suspect quite a few readers are already thinking: how do I apply this to my own model? Can I fine-tune it on my own task?

Sorry to break some rather pessimistic news: I tried fine-tuning this ~1.5-billion-parameter GPT2 with the Adam optimizer on a 22GB TITAN RTX at my company, and found that even a batch size of 1 wouldn't run... In the end, I discovered that only with the AdaFactor optimizer could I get fine-tuning to actually work.

I'll write a separate post about AdaFactor when I get the chance.