Open-sourcing a Keras implementation of the DGCNN reading-comprehension QA model

开源一版DGCNN阅读理解问答模型（Keras版） 苏剑林 · 2019-08-20 · Machine Learning model Q&A

Last year I wrote A CNN-based reading-comprehension QA model: DGCNN, introducing a simple, purely convolutional QA model. At the time it was implemented in TensorFlow and never released as open source. A few days ago I found some spare time to reimplement it in Keras, and I've decided to open-source it now.

Overview of the model

I won't repeat the basic introduction to DGCNN here. The model presented in this post is not simply a re-implementation of the earlier one — it has been modified in several respects, which I'll describe below.

1. The offline validation score of the model released here is roughly 0.72 (compared to about 0.75 previously);
2. This version operates at the character level, using the "hybrid character-word embedding" technique I explored earlier (the previous version operated at the word level);
3. All hand-crafted features have been removed (the previous version used 8 of them);
4. Positional embeddings have been removed (previously they were concatenated onto the input);
5. The model architecture and training details have been tweaked slightly.

Working at the character level makes labeling more flexible (avoiding tokenization errors); removing the hand-crafted features also increases the model's flexibility and speeds up inference. As for dropping the positional embeddings, several experiments showed they didn't yield any noticeable improvement. Other adjustments include training with the newly released RAdam optimizer, among other things.

This release isn't aimed at squeezing out the highest possible score — it's simply meant to provide a Keras version as a reference for others. I believe there's still plenty of room for improvement, so anyone interested is welcome to tune it further (both the code and the dataset are publicly available).

Open-source repository

GitHub: https://github.com/bojone/dgcnn_for_reading_comprehension

(Environment: Python 2.7 + TensorFlow 1.8 + Keras 2.2.4. Please don't ask about environment setup issues — thanks!)

Word embeddings: https://pan.baidu.com/s/1YYE2T3f-lPyLBrJuUowAsA, password: 5p0h

Dataset: https://pan.baidu.com/s/11C21BAupOpiYWoOx23J7Mg, password: dh9w

(If there is anything improper about releasing this dataset, please let me know and I will remove it as soon as possible.)

Final words

Enjoy trying it out, and feel free to discuss further~