The universal seq2seq: reading comprehension QA based on seq2seq

万能的seq2seq：基于seq2seq的阅读理解问答 苏剑林 · 2019-12-05 · Machine Learning Q&A language model text generation

Today I've added another example to bert4keras: reading-comprehension-style question answering (task_reading_comprehension_by_seq2seq.py), using the same corpora as before, namely WebQA and SogouQA, with a final score of around 0.77 (single model, no careful tuning).

Brief description of the method

Since the main purpose this time is to add a demo to bert4keras, efficiency isn't really the primary concern. The goal here is mainly generality and ease of use, so I've used the most universal approach available — seq2seq — to do reading comprehension.

With seq2seq, you basically don't need to worry much about model design: just concatenate the passage and the question, then predict the answer. Beyond that, the seq2seq approach naturally provides a way to judge whether a passage contains an answer at all, and it also naturally leads to a strategy for voting across multiple passages. All in all, ignoring efficiency, seq2seq is a rather elegant way of doing reading comprehension.

To implement seq2seq here I've again used the UNILM approach. If you're not familiar with it, you can first read From Language Models to Seq2Seq: Transformer as Theater, All Thanks to Masking for the relevant background. more

Model details

Building a seq2seq model with the UNILM approach is basically a one-liner in bert4keras, so the main work in this example isn't really in building the model, but in handling the input and output.

Input format

First, the input. The input format is very simple, and one diagram makes it clear:

Diagram of the seq2seq model for reading comprehension

Output processing

If you feed in a single passage and a single question to answer, you can just decode in the usual seq2seq way — i.e., beam search.

However, WebQA and SogouQA are aimed at search scenarios, where multiple articles may exist simultaneously to answer the same question. This brings up the question of how to choose a voting scheme. One naive idea is: for each passage, decode independently with the question via beam search, giving a confidence score for each answer, and then vote following the scheme in A CNN-based reading-comprehension QA model: DGCNN. The difficulty with this approach lies in coming up with a reasonable confidence score for each answer; compared with the approach we'll describe next, it feels less natural, and it's also somewhat less efficient.

Here we present a scheme that "meshes" more naturally with beam search:

First exclude passages that have no answer, then, at each step of decoding the answer, average the probability values predicted by all passages (in a certain way).

Specifically, each passage is concatenated with the question, and each gives its own probability distribution for the first character. Passages whose first predicted character is [SEP] are deemed to have no answer, and are excluded. After exclusion, the probability distributions for the first character of the remaining passages are averaged, and then the top-k candidates are kept (standard beam search procedure). When predicting the second character, each passage is combined with each of the top-k candidates to predict its own probability distribution for the second character; these are again averaged over passages, and the top-k are kept. This continues until [SEP] appears. (It's essentially ordinary beam search plus averaging across passages. If this description isn't clear enough, you'll just have to check the source code.)

Furthermore, there should be two ways of generating the answer: one is extractive, where the answer must be a span from the passage, and the other is generative, where there's no need to worry about whether the answer is a span of the passage — you just decode and generate the answer directly. Both approaches are handled with corresponding logic in the decoding code in this article.

Experimental code

Code link: task_reading_comprehension_by_seq2seq.py

Finally, on SogouQA's own evaluation script, the score on the validation set is about 0.77 (Accuracy=0.7259005836184343, F1=0.813860036706151, Final=0.7698803101622926). The single-model result far exceeds that of the earlier Open-sourcing a DGCNN reading-comprehension QA model (Keras version) model. Of course, this improvement comes at a cost — the prediction speed is much lower, managing only about 2 samples per second.

(The model hasn't been carefully tuned, so there's presumably still room for improvement; for now it's mainly meant as a demo.)

Summary

This post mainly presented a reading comprehension example based on BERT and the seq2seq approach, and introduced a beam search strategy for voting across multiple passages, for readers' reference and experimentation.