A Poetry-Writing Robot Based on CNN and VAE: Composing Poems from Random Noise

基于CNN和VAE的作诗机器人：随机成诗 苏剑林 · 2018-03-24 · Machine Learning literature VAE generative model text generation

A few days ago I wrote an accessible introduction to VAEs, which was well received by some readers. But aren't you tired of every introduction only offering an MNIST-level demo? Don't worry—here comes a more classic VAE toy: a poetry-writing robot.

Why "more classic"? In the previous post we noted that images generated by VAEs tend to be blurrier than those generated by GANs—that is, on the battlefield of image generation, VAE is at a disadvantage. However, in the field of text generation, VAE wins handsomely. This is because GANs hope to train a discriminator (a metric) directly, but for text this metric is very likely discrete and non-differentiable, which makes pure GANs hard to train. VAEs don't have this step—they work by reconstructing the input, and this reconstruction process works equally well for images and for text. So for VAE, text generation is just as basic and direct an application as image generation; whereas for (current) GANs, it remains a difficult, persistent "sore spot."

Well, in ancient times Cao Zhi composed a poem in seven steps; today, VAE composes poems at random. Let's get started~

Model

For many people, poetry is a wonderful thing—wonderful precisely because most people don't really understand poetry, yet everyone has some vague sense of what a poem should look like. So as long as the generated "poem" looks roughly the part, we usually conclude that the robot can write poetry. Hence a "poetry-writing robot" is a purely playful toy: being able to produce a few lines of verse doesn't mean much about how good the model's general language generation ability is, nor does it reflect any deep understanding of NLP.

CNN + VAE

As far as this toy is concerned, the model is actually fairly simple—it mainly combines a 1D CNN with a VAE. Since the length of the generated poem is fixed, I used pure CNNs for both the encoder and the decoder. The model architecture looks roughly like this:

CNN + VAE poem generation modelmore

Specifically, each character is first embedded into a vector, then stacked CNN layers are used for encoding, followed by pooling to get the encoder's output. From this output, we compute a mean and a variance, sample from the resulting normal distribution, and proceed to decoding. Since at this point we only have a single encoder output, but the final output must consist of multiple characters, several different fully-connected layers are applied first to produce diverse outputs, followed by further fully-connected layers.

GCNN

The CNN used here is not an ordinary CNN+ReLU, but the GCNN proposed by Facebook. It essentially runs two separate CNNs of identical shape—one without any activation function, and one activated with sigmoid—and then multiplies the two results together. The sigmoid branch thus effectively acts as a "gate."

I first encountered GCNN in the paper Language Modeling with Gated Convolutional Networks, and then saw it again in Convolutional Sequence to Sequence Learning. I also gave a brief introduction to it in my own Sharing a slide: Fancy Natural Language Processing.

Empirically, GCNN clearly outperforms ordinary CNN+ReLU on many NLP tasks.

Experiments

The experiments were done in Python 2.7 and Keras (with a TensorFlow backend)~

Code

With the discussion above, and combined with the VAE example bundled with Keras, implementing the whole model isn't difficult at all. Since this is purely for demonstration, I chose the simplest case: five-character-per-line poems, and rather than requiring a full poem to be generated, only a single line (10 characters) is produced—making it more like composing a couplet, really.

Code: https://github.com/bojone/vae/blob/master/vae_shi.py

The training corpus is the complete collection of Tang poetry, which has also been uploaded to GitHub. The model hasn't been thoroughly tuned, so I'll leave that for interested readers to tinker with and improve.

Training

To observe how the generated lines change over the course of training, I wrote an evaluator, whose output looks like this:

Poetry robot training process

As you can see, the quality of the generated lines does improve as training progresses.

Testing

Below are some lines randomly generated by the trained model. Strictly speaking they're nothing special—after all, this is just a mapping from random numbers to lines of verse, poems produced purely by chance~ Still, you can see that these "lines" look reasonably plausible in terms of parallelism and tonal pattern.

出上无花客，相来一日时。
从瞻大车策，萧盖偃车矛。
今见青衣去，萧凉白叶风。
帝城今不战，征罪在天兵。
鹤仰临山里，逶留出太回。
画关斜过水，残色迥过杨。
上道皆有战，四门不如兵。
涧烟含雨沥，风影动风风。
登回一落景，一处更相期。
芳酒不无醉，长楼酒更春。
天月满云管，楚女长南闻。
明今今有矣，不得道无生。
朝明开绿菊，香影下红枝。
世外相扁处，逍遥无旧目。
唯闻含玉色，讵见月光光。
春风将乐节，风服未谁娱。
万年君何在，今年酒未新。
回辔参相召，乘歌会使程。
今有千人醉，不然一子心。
今风泛云会，清日白衣新。
Note: since the demo model only generates single lines, there is no connection between the different lines listed above.

Final Remarks

The point of this experiment was not to produce especially high-quality poetry, but to demonstrate VAE-based text generation. There are similar poetry-writing robots online already, though most popular ones are based on an "RNN + language model" approach, generally requiring some seed words as input to complete a poem. VAE, on the other hand, truly achieves the process of mapping random noise directly into lines of verse~