Sohu Text Matching: A Multi-task Baseline Based on Conditional LayerNorm

搜狐文本匹配：基于条件LayerNorm的多任务baseline 苏剑林 · 2021-04-16 · Machine Learning language model competition semantic similarity

A while back I came across the "2021 Sohu Campus Text Matching Algorithm Competition" and found the task quite interesting, so I gave it a try. However, since the competition is only open to enrolled students, I couldn't participate as an official contestant. So instead, I'm open-sourcing my approach as a baseline for the competition, for everyone's reference.

GitHub link: https://github.com/bojone/sohu2021-baseline

Task Introduction

As the name suggests, the competition's task is text matching, i.e., determining whether two pieces of text are similar — normally a fairly standard task. What makes it interesting, though, is that it's split into multiple subtasks. Specifically, the task is divided into two major categories, A and B, where category A uses a looser matching standard and category B a stricter one. Each of these two categories is further split into three subtypes: "short-short matching", "short-long matching", and "long-long matching". So although the task type is nominally the same throughout, strictly speaking there are six distinct subtasks.more

# A类样本示例
{
    "source": "英国伦敦，20/21赛季英超第20轮，托特纳姆热刺VS利物浦。热刺本赛季18轮联赛是9胜6平3负，目前积33分排名联赛第5位。利物浦本赛季19轮联赛是9胜7平3负，目前积34分排名联赛第4位。从目前的走势来看，本场比赛从热刺的角度来讲，是非常被动的。最终，本场比赛的比分为托特纳姆热刺1-3利",
    "target": " 北京时间1月29日凌晨4时，英超联赛第20轮迎来一场强强对话，热刺坐镇主场迎战利物浦。  热刺vs利物浦，比赛看点如下： 第一：热刺能否成功复仇？双方首回合，热刺客场1-2被利物浦绝杀，赛后穆里尼奥称最好的球队输了，本轮热刺主场迎战利物浦，借着红军5轮不胜的低迷状态，能否成功复仇？ 第二：利物浦近",
    "labelA": "1"
}

# B类样本示例
{
    "source": "英国伦敦，20/21赛季英超第20轮，托特纳姆热刺VS利物浦。热刺本赛季18轮联赛是9胜6平3负，目前积33分排名联赛第5位。利物浦本赛季19轮联赛是9胜7平3负，目前积34分排名联赛第4位。从目前的走势来看，本场比赛从热刺的角度来讲，是非常被动的。最终，本场比赛的比分为托特纳姆热刺1-3利",
    "target": " 北京时间1月29日凌晨4时，英超联赛第20轮迎来一场强强对话，热刺坐镇主场迎战利物浦。  热刺vs利物浦，比赛看点如下： 第一：热刺能否成功复仇？双方首回合，热刺客场1-2被利物浦绝杀，赛后穆里尼奥称最好的球队输了，本轮热刺主场迎战利物浦，借着红军5轮不胜的低迷状态，能否成功复仇？ 第二：利物浦近",
    "labelB": "0"
}

Generally speaking, completing this task requires at least two models, since categories A and B have different classification standards; and if we want to be even more fine-grained, we'd end up needing six separate models. The problem is that training six models independently tends to be laborious, and the different tasks can't benefit from each other to improve performance. So naturally, we should think of sharing part of the parameters and turning this into a multi-task learning problem.

Model Overview

Of course, if we treat this as an ordinary multi-task learning problem, that would be too generic an approach. Given the characteristic that these tasks are "identical in form but different in standard," I came up with the idea of using Conditional Layer Normalization to handle all six subtasks with a single model.

We've already introduced conditional LayerNorm before, in the article Conditional Text Generation Based on Conditional Layer Normalization. Although the example used there was text generation, its applicability is not limited to that scenario. Simply put, conditional LayerNorm is a scheme for injecting a condition vector into a Transformer to control its output, by incorporating the condition into the $\beta,\gamma$ of the LayerNorm layer.

For the six tasks in this competition, we only need to feed the task type into the model as a condition, and then a single model can handle all six different tasks. Here's an illustration:

Using conditional LayerNorm to handle multiple similar tasks

In this way, the entire model is shared; the only difference is that, at input time, the task type id is fed in alongside the input sentence, achieving maximal parameter sharing.

Reference Code

The implementation of conditional LayerNorm has long been built into bert4keras, so once I settled on this design, implementing it with bert4keras was a natural next step. The reference code is here:

GitHub link: https://github.com/bojone/sohu2021-baseline

The code uses RoFormer as the base model. The main consideration here is that in "long-long matching," the total length after concatenating the two texts can be quite long — using a word-based RoFormer can shorten the sequence length, allowing longer texts to be processed under the same computational budget. Moreover, the RoPE positional encoding used by RoFormer can in principle handle text of arbitrary length. I ran the code a few times and got an offline F1 of around 0.74, with an online test-set F1 of roughly 0.73 after submission. Tested on a 3090, a single epoch takes about an hour, and running 4-5 epochs is roughly enough.

The current code mixes all the data together and trains on it randomly, which has a minor drawback: samples that originally had shorter sequence lengths also get padded to the maximum length during training, slowing down training for short-sequence samples (though it's still certainly faster than training six separate models). One possible optimization would be to batch samples with similar lengths together, but I'm too lazy to implement that — I'll leave it as an exercise for you to optimize.

Summary

This post shared a baseline for the Sohu text matching competition, mainly using conditional LayerNorm to increase the model's diversity, so that a single model can handle different types of data and produce different corresponding outputs.