Jumping on the bandwagon to play with the largest Chinese GPT2 model currently available (bert4keras)

跟风玩玩目前最大的中文GPT2模型（bert4keras） 苏剑林 · 2020-11-20 · Machine Learning language model text generation attention

I'm sure many readers have already seen that Tsinghua University and the Beijing Academy of Artificial Intelligence (BAAI) recently launched the "Qingyuan Project" (see the related article "Has the Chinese Version of GPT-3 Arrived? BAAI Releases Qingyuan CPM — a Large-Scale Pretrained Model Centered on Chinese"), which open-sourced CPM-LM, currently the largest Chinese GPT2 model (2.6 billion parameters). It's said that models with 20 billion or even 100 billion parameters will be open-sourced in the future, with the goal of building "the GPT3 of the Chinese-speaking world."

Official demo image of CPM-LM's Few-Shot performance

As we know, GPT3 can achieve Few-Shot performance without any fine-tuning, and in the demo examples currently released for CPM-LM, the Few-Shot results are also quite impressive, which makes people eager to try it out — myself included. Naturally, if I wanted to try it, I had to adapt it to my own bert4keras to make things convenient, so the adaptation work began. I thought this would be an easy task, but it turned out to take nearly three days of stumbling through pitfalls before I got it working. Let me record the process of pitfalls and testing here. more

Model Introduction

The first model released under this project is called CPM-LM, with roughly 2.6 billion parameters, pretrained on 100GB of Chinese text data. It's a unidirectional language model; readers can check the details themselves via the links below. With such a huge parameter count, we generally just use it directly rather than fine-tuning it. Its core capability is unconditional random text generation, though of course we can also provide it with some guidance and use it for text continuation. As for applications like Few-Shot learning, these are essentially just variants of text continuation.

Homepage: https://cpm.baai.ac.cn/
GitHub: https://github.com/TsinghuaAI/CPM-Generate
WeChat public account: https://mp.weixin.qq.com/s/oI2Ak-M57MSuycLVpVEiHw

Let me talk about the model architecture issue here, which was the first pitfall I fell into during adaptation. CPM-LM's model architecture is the same as OpenAI's GPT2, so to put it plainly, this is just a 2.6-billion-parameter Chinese GPT2 model. At first I didn't look carefully, and was slightly misled by the CPM-LM-TF2 project, which led me to assume early on that its structure was the same as GPT2_ML (GPT2_ML is neither GPT nor GPT2, but something in between the two). For a long while I couldn't get reasonable results. Once I realized the issue, rebuilding the GPT2 model and adapting the corresponding weights wasn't much of a challenge anymore, including converting the weights to tf format — with the CPM-LM-TF2 project as reference, it wasn't too difficult either.

Tokenizer

The second pitfall I ran into during adaptation concerned the tokenizer. I have to say, in my view the tokenizer written for CPM-LM is really quite crude, and it still bothers me to this day.

This tokenizer is actually just a wrapper around Google's sentencepiece, but it's wrapped in a particularly inelegant way — a nightmare for anyone with a touch of OCD. Specifically, tokenization tools like BERT's tokenizer or sentencepiece by default strip out separators such as spaces and newlines. But CPM-LM wanted to preserve spaces and newlines, so before feeding text into the tokenizer, it replaces them with other symbols (currently spaces are replaced with "▂" and newlines with "▃"), and then replaces them back after the output. This is a common approach and understandable. But what I really cannot understand is that the replacement symbol for newlines, "▃", isn't even in the vocabulary of its sentencepiece model! To avoid "▃" turning into <unk>, CPM-LM replaces it again with <cls> — that is, it performs a second substitution just to get the id for the newline character...

The first time I saw a design like this, I was honestly on the verge of a breakdown internally: is it really that hard to just add one extra character to sentencepiece? Why did it have to be written this way... But there's no way around it — whoever open-sources the model is the boss, so I just had to figure out how to adapt to it. After thinking about it for a very long time, I patched and tweaked bert4keras's original SpTokenizer, and finally, with some effort, got it working.

Usage Test

That's enough complaining for now. In short, after more than two days of tinkering, starting from version 0.9.3, bert4keras can now load the CPM-LM model. Just running prediction alone requires an estimated 16GB+ of GPU memory (I myself have a 22GB RTX). The weight conversion process and the basic loading setup can be found here:

GitHub: https://github.com/bojone/CPM_LM_bert4keras

Some Few-Shot results (the outputs have a degree of randomness; if you only care about Few-Shot performance, you might consider switching the decoding method to beam search):

# 常识推理
# 本例输出：北京
query = u"""
美国的首都是华盛顿
法国的首都是巴黎
日本的首都是东京
中国的首都是
"""
print(text_expansion.generate(query[1:-1], 1)[0])

# 单词翻译
# 本例输出：bird
query = u"""
狗 dog
猫 cat
猪 pig
鸟 
"""
print(text_expansion.generate(query[1:-1], 1)[0])

# 主语抽取
# 本例输出：杨振宁
query = u"""
从1931年起，华罗庚在清华大学边学习边工作 华罗庚
在一间简陋的房间里，陈景润攻克了“哥德巴赫猜想” 陈景润
在这里，丘成桐得到IBM奖学金 丘成桐
杨振宁在粒子物理学、统计力学和凝聚态物理等领域作出里程碑性贡献 
"""
print(text_expansion.generate(query[1:-1], 1)[0])

# 三元组抽取
# 本例输出：张红,体重,140斤
query = u"""
姚明的身高是211cm，是很多人心目中的偶像。 ->姚明，身高，211cm
虽然周杰伦在欧洲办的婚礼，但是他是土生土长的中国人->周杰伦，国籍，中国
小明出生于武汉，但是却不喜欢在武汉生成，长大后去了北京。->小明，出生地，武汉
吴亦凡是很多人的偶像，但是他却是加拿大人，另很多人失望->吴亦凡，国籍，加拿大
武耀的生日在5月8号，这一天，大家都为他庆祝了生日->武耀，生日，5月8号
《青花瓷》是周杰伦最得意的一首歌。->周杰伦，作品，《青花瓷》
北京是中国的首都。->中国，首都，北京
蒋碧的家乡在盘龙城，毕业后去了深圳工作。->蒋碧，籍贯，盘龙城
上周我们和王立一起去了他的家乡云南玩昨天才回到了武汉。->王立，籍贯，云南
昨天11月17号，我和朋友一起去了海底捞，期间服务员为我的朋友刘章庆祝了生日。->刘章，生日，11月17号
张红的体重达到了140斤，她很苦恼。->
"""
print(text_expansion.generate(query[1:-1], 1)[0])

Summary

This article gave a brief introduction to CPM-LM, a newly open-sourced 2.6-billion-parameter GPT2 model from Tsinghua University, and described how I adapted it into the bert4keras framework, complaining a bit along the way about the pitfalls encountered during conversion. Finally, it showcased CPM-LM's rather impressive Few-Shot performance.