I built my own bert4keras

自己实现了一个bert4keras 苏剑林 · 2019-08-27 · Machine Learning language model NLP Keras attention

Sharing a personal implementation of bert4keras:

https://github.com/bojone/bert4keras

This is my own reimplementation of BERT in Keras, aimed at achieving a Keras-based BERT with code that's as clean as possible.

Notes

At this point BERT is basically fully implemented, and it can successfully load the official pretrained weights — verified to give outputs identical to keras-bert, so feel free to use it with confidence.

The original motivation for this project was to make modification and customization easier, so it may get updated fairly frequently.

Stars are welcome, but forking isn't recommended, since whatever version you fork might become outdated pretty quickly.

Usage

Quick install:

pip install git+https://www.github.com/bojone/bert4keras.git

Sample code:

#! -*- coding: utf-8 -*-
# 测试代码可用性

from bert4keras.models import build_transformer_model
from bert4keras.tokenizers import Tokenizer
import numpy as np

config_path = '../../kg/bert/chinese_L-12_H-768_A-12/bert_config.json'
checkpoint_path = '../../kg/bert/chinese_L-12_H-768_A-12/bert_model.ckpt'
dict_path = '../../kg/bert/chinese_L-12_H-768_A-12/vocab.txt'

tokenizer = Tokenizer(dict_path) # 建立分词器
model = build_transformer_model(config_path, checkpoint_path) # 建立模型，加载权重

# 编码测试
token_ids, segment_ids = tokenizer.encode(u'语言模型')
print(model.predict([np.array([token_ids]), np.array([segment_ids])]))

The examples I gave earlier based on keras-bert in When BERT Meets Keras: Perhaps the Simplest Way to Get Started with BERT still apply to this project — you just need to switch how the base_model is loaded to this project's approach.

Currently only Python 2.7 is guaranteed to be supported; the experimental environment is TensorFlow 1.8+ and Keras 2.2.4+.

(A friend has tested it and said Python 3 also works directly without errors, so Python 3 users are welcome to give it a try. But I haven't tested it myself, so I can't guarantee it.)

Of course, if anyone who enjoys contributing finds any bugs, feel free to point them out, or even submit pull requests~

Background

I had previously always used the excellent keras-bert by CyberZHG. If all you need is to call and fine-tune BERT within Keras, keras-bert is already more than satisfactory.

However, if you want to load the official pretrained weights and then modify BERT's internal structure, keras-bert becomes rather difficult to accommodate, because for the sake of code reuse it wraps almost every small module into its own separate library — for instance, keras-bert depends on keras-transformer, keras-transformer depends on keras-multi-head, and keras-multi-head depends on keras-self-attention. With this chain of dependencies piling up, making changes becomes quite a headache.

So I decided to rewrite a Keras version of BERT from scratch, trying to implement it completely within just a few files, cutting down on these dependencies while still retaining the ability to load the official pretrained weights.

Acknowledgements

Thanks to CyberZHG for implementing keras-bert — this implementation borrows quite a bit from keras-bert's source code, and I sincerely appreciate this generous, selfless contribution.