Conditional Text Generation Based on Conditional Layer Normalization

基于Conditional Layer Normalization的条件文本生成 苏剑林 · 2019-12-14 · Machine Learning language model text generation attention

From the article From Language Model to Seq2Seq: With the Right Mask, Transformers Can Do Anything], we know that with an appropriate attention mask, Bert (or other Transformer models) can be used for unconditional generation (language modeling) as well as sequence translation (Seq2Seq) tasks.

But what about conditional generation? For example, controlling the category of the text and generating text randomly by category — that is, a Conditional Language Model; or passing in an image and generating a piece of related text description — that is, Image Captioning.

Related Work

An August paper, Encoder-Agnostic Adaptation for Conditional Language Generation], gives a fairly systematic analysis of several schemes for using pretrained models for conditional generation. A September paper, CTRL: A Conditional Transformer Language Model for Controllable Generation], provides a model pretrained on conditional generation — though at its core it's still a language model just like GPT, and can only take text as its condition. And a more recent paper, Plug and Play Language Models: a Simple Approach to Controlled Text Generation], converts $p(x|y)$ into $p(x)p(y|x)$ to explore conditional generation based on pretrained models.

However, none of these classic pieces of work are what this article is about. This article focuses on the scenario of text generation conditioned on a fixed-length vector, using a method called Conditional Layer Normalization — which incorporates the condition into the $\beta$ and $\gamma$ of Layer Normalization. more

The Idea in Detail

The idea behind Conditional Layer Normalization comes from the popular approach used in conditional GANs for images — Conditional Batch Normalization (Conditional BN). Related material can be found in From DCGAN to SELF-MOD: An Overview of the Evolution of GAN Architectures]. Conditional BN also has a variant called AdaIN (Adaptive Instance Normalization). Both Conditional BN and AdaIN turn the $\beta$ and $\gamma$ in existing normalization methods into functions of the input condition, thereby allowing the condition to control the behavior of the generation.

In Transformer models like Bert, the main normalization method is Layer Normalization, so it's natural to think of turning the corresponding $\beta$ and $\gamma$ into functions of the input condition, in order to control the generation behavior of the Transformer model — this is the core idea behind Conditional Layer Normalization. (I haven't yet seen any other work following the same line of thinking, so this counts as something I cooked up on my own behind closed doors.)

Conditional Normalization illustration

For an already pretrained model, there already exist unconditional $\beta$ and $\gamma$, both of which are fixed-length vectors. We can use two different transformation matrices to project the input condition to the same dimensionality as $\beta,\gamma$, and then add the two transformed results to $\beta$ and $\gamma$ respectively. To avoid disrupting the original pretrained weights, both transformation matrices can be initialized to all zeros (a single-layer network can be initialized to all zeros, though consecutive multi-layer networks should not be initialized this way). This way, in the initial state the model still stays consistent with the original pretrained model.

Code Implementation

Intuitively, this kind of fine-tuning aimed at text generation should benefit more from using an autoregressive pretrained model like GPT. But in fact, as the previous article From Language Model to Seq2Seq: With the Right Mask, Transformers Can Do Anything] already showed, even loading Bert's pretrained weights for a generation task still performs well. So regardless of which Transformer-based pretrained model you use, it can be considered for fine-tuning into a text generation model. This article still uses pretrained Bert as the base model for its experiments.

As for the code, the Conditional Layer Normalization trick described in this article has already been integrated into bert4keras], which I develop. The base function build_transformer_model now has the following new parameters:

1. layer_norm_cond: if this parameter is not None, it means it is a tensor with shape=[batch_size, cond_size], used as the condition for Layer Normalization;
2. layer_norm_cond_size: if this parameter is not None and layer_norm_cond is None, it means it is an integer, and an input layer with shape=[batch_size, layer_norm_cond_size] will be built automatically as the condition for Layer Normalization;
3. layer_norm_cond_hidden_size: if this parameter is not None, it means it is an integer used to first project the input condition to a lower-dimensional space. This is because the input condition may have very high dimensionality, and projecting it directly to hidden_size (e.g. 768) could involve too many parameters, so it can first be projected to a lower-dimensional space and then projected back up;
4. layer_norm_cond_hidden_act: the activation function used when projecting to the lower-dimensional space; if None, no activation function is applied (linear activation);
5. additional_input_layers: additional input layers. If a tensor is passed in externally as a condition, then all the input layers that the condition tensor depends on need to be added here as input layers, so that the final model can be built correctly.

Experimental Results

No amount of explanation is as useful as actually seeing some examples. I ran two experiments to validate the effectiveness of Conditional Layer Normalization. One controls text generation via sentiment polarity — essentially the inverse problem of sentiment classification — where the class embedding is used directly as the condition for Layer Normalization. The other is Image Captioning, where a pretrained ImageNet model encodes an image into a fixed-length vector used as the condition for Layer Normalization.

The code for these two experiments is in task_conditional_language_model.py] and task_image_caption.py] respectively.

Sentiment Text Generation

For sentiment text generation, the training set used is the sentiment classification corpus] I collected and organized previously, simply with the input text and label roles reversed. At generation time, sampling is done randomly according to probability, allowing different pieces of text to be generated.

Some sample outputs:

Positive samples:
[u'外观时尚、漂亮、性价比高。', u'外观漂亮，配置均衡，比较满意，性价比高，外观漂亮，性能较高。', u'我是在大学的时候看到这本书的，所以一直在买。书中的作者是林静蕾，她用自己的口吻写出了一个孩子成长中的心路历程，让我看到了她们成长中的不同之处，以及她们成长过程中的不同境界。让我很欣赏！', u'我想这是一本能够告诉读者什么是坏的，而不是教你怎样说话，告诉我什么是错。这里我推荐了《我要讲故事》，这本书是我很喜欢的一本书，我认为它的理由很多，但是，我相信我。如果你从中得到一些改进，或者你已经有了一个明智的决定。', u'我们一家五口住的是标间，大床房，大床的床很舒服；而我们在携程网上订了两套大床房，这个酒店的价格还是比较合理的；但是房间的隔音效果不太理想，有点响的声音；酒店门口的地铁在施工中，不方便；但是酒店的门口的出租车不知道是哪个车的，打车不是很方便；酒店外面的停']
Negative samples:
[u'不知道是不是因为电池不太好，不是我不喜欢。', u'看了评论才买的. 结果发现不是那么便宜, 价格也不便宜.', u'1、外壳不容易沾手印，不容易洗洗2、屏幕有点旧，不能下载铃声', u'我是7月6日订购了《杜拉拉升职记》并已通过银行付款，为什么订单下了两周多至今还未到货？是收货时间太快了，可能就这么过去了吧？', u'这本书我是在网上先看了一遍，后来我再看了一遍。感觉作者的文笔实在太烂了，特别是在写他的博客时特别别扭，写得很不专业，特别是他写股票时那个情绪调节的小男孩，简直就是自作聪明的样子，简直就是自作聪明的一种表现！']

Image Caption

For Image Captioning, I used the COCO dataset] as an example, since its images cover a richer variety of scenes. There was also a Chinese image captioning competition] hosted by challenger.ai in 2017, which included a decent dataset as well (readers can find a way to collect it themselves), though its images tend to be somewhat more monotonous in scene variety.

Some sample outputs:

Model prediction: a baseball game in progress with the batter up to plate.

Model prediction: a train that is sitting on the tracks.

image_id: COCO_val2014_000000524611.jpg
url: http://images.cocodataset.org/val2014/COCO_val2014_000000524611.jpg]
predict: a train that is sitting on the tracks.
references: [u'A train carrying chemical tanks traveling past a water tower.', u'Dual train tracks with a train on one of them and a water tower in the background.', u'a train some trees and a water tower ', u'Train on tracks with water tower for Davis Junction in the rear.', u'A train on a train track going through a bunch of trees.']
image_id: COCO_val2014_000000202923.jpg
url: http://images.cocodataset.org/val2014/COCO_val2014_000000202923.jpg]
predict: a baseball game in progress with the batter up to plate.
references: [u'Batter, catcher, and umpire anticipating the next pitch.', u'A baseball player holding a baseball bat in the game.', u'A baseball player stands ready at the plate.', u'Baseball players on the field ready for the pitch.', u'A view from behind a mesh fence of a baseball game.']

Summary

This article proposes the idea of using Conditional Layer Normalization to incorporate external conditions into a pretrained model. Its most direct application is conditional text generation, but it's not limited to generative models — it can also be used in classification models and other scenarios (where the external condition might be information from another modality, used to assist classification). Finally, based on bert4keras, a code implementation and two examples were given.