Notes on a Semi-Supervised Sentiment Analysis Attempt

记录一次半监督的情感分析 苏剑林 · 2017-05-04 · Machine Learning deep learning text mining

This post describes a not-entirely-successful attempt at semi-supervised learning: on the IMDB dataset, I trained a text sentiment classification model using 1,000 randomly sampled labeled examples, and tested it on the remaining 49,000 examples, achieving a test accuracy of 73.48%.

Motivation

The idea for this post comes from an OpenAI article:

Quotes

The article describes a method for unsupervised (actually semi-supervised) sentiment classification that achieved excellent experimental results. However, the experiments in the article are massive in scale and essentially impossible for an individual to reproduce (training took a month on 4 Pascal GPUs). That said, the underlying idea is quite simple, and based on it we can put together a "budget version." Here's the idea:

When we do sentiment classification with deep learning, the usual approach is Embedding layer + LSTM layer + Dense layer (sigmoid activation). What we normally call word embeddings amounts to pretraining the Embedding layer (which has the largest number of parameters and is most prone to overfitting). OpenAI's insight was: why not pretrain the LSTM layer as well? The pretraining method is again a language model. Of course, to make sure the pretrained representations don't lose sentiment information, the LSTM's hidden layer needs to be reasonably large.

If the LSTM layer is also pretrained, then the remaining Dense layer has very few parameters, and so it can be trained to a reasonable degree with just a small number of labeled samples. That's the whole idea behind this semi-supervised approach. As for the "sentiment neuron" that OpenAI's article talks about — that's really just a vivid way of describing things.

Of course, judging purely from the sentiment analysis task itself, my 73.48% accuracy is nothing to write home about — a plain "dictionary + rules" approach can easily exceed 80% accuracy. I'm simply verifying the feasibility of this experimental approach. I believe that if the scale could match OpenAI's, the results would be much better. Moreover, what this post really wants to convey is a modeling strategy, not something limited to sentiment analysis — the same idea can be applied to any binary or even multi-class classification problem.

The Process

First, load the dataset and re-split it into training and test sets:

from keras.preprocessing import sequence
from keras.models import Model
from keras.layers import Input, Embedding, LSTM, Dense, Dropout
from keras.datasets import imdb
from keras import backend as K
import numpy as np

max_features = 10000 #保留前max_features个词
maxlen = 100 #填充/阶段到100词
batch_size = 1000
nb_grams = 10 #训练一个10-gram的语言模型
nb_train = 1000 #训练样本数

#加载内置的IMDB数据集
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_lm_ = np.append(x_train, x_test)

#构造用来训练语言模型的数据
#这里只用了已有数据，实际环境中，可以补充其他数据使得训练更加充分
x_lm = []
y_lm = []
for x in x_lm_:
		for i in range(len(x)):
			x_lm.append([0]*(nb_grams - i + max(0,i-nb_grams))+x[max(0,i-nb_grams):i])
			y_lm.append([x[i]])

x_lm = np.array(x_lm)
y_lm = np.array(y_lm)
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
x = np.vstack([x_train, x_test])
y = np.hstack([y_train, y_test])

#重新划分训练集和测试集
#合并原来的训练集和测试集，随机挑选1000个样本，作为新的训练集，剩下为测试集
idx = range(len(x))
np.random.shuffle(idx)
x_train = x[idx[:nb_train]]
y_train = y[idx[:nb_train]]
x_test = x[idx[nb_train:]]
y_test = y[idx[nb_train:]]

Then build the model:

embedded_size = 100 #词向量维度
hidden_size = 1000 #LSTM的维度，可以理解为编码后的句向量维度。

#encoder部分
inputs = Input(shape=(None,), dtype='int32')
embedded = Embedding(max_features, embedded_size)(inputs)
lstm = LSTM(hidden_size)(embedded)
encoder = Model(inputs=inputs, outputs=lstm)

#完全用ngram模型训练encode部分
input_grams = Input(shape=(nb_grams,), dtype='int32')
encoded_grams = encoder(input_grams)
softmax = Dense(max_features, activation='softmax')(encoded_grams)
lm = Model(inputs=input_grams, outputs=softmax)
lm.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
#用sparse交叉熵，可以不用事先将类别转换为one hot形式。

#情感分析部分
#固定encoder，后面接一个简单的Dense层（相当于逻辑回归）
#这时候训练的只有hidden_size+1=1001个参数
#因此理论上来说，少量标注样本就可以训练充分
for layer in encoder.layers:
    layer.trainable=False

sentence = Input(shape=(maxlen,), dtype='int32')
encoded_sentence = encoder(sentence)
sigmoid = Dense(10, activation='relu')(encoded_sentence)
sigmoid = Dropout(0.5)(sigmoid)
sigmoid = Dense(1, activation='sigmoid')(sigmoid)
model = Model(inputs=sentence, outputs=sigmoid)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Now, train the language model — this part is fairly time-consuming:

#Training the language model is time-consuming; generally a couple of epochs is enough

lm.fit(x_lm, y_lm,
       batch_size=batch_size,
       epochs=3)

The language model training results were:

Epoch 1/3
11737946/11737946 [==============================] - 2400s - loss: 5.0376
Epoch 2/3
11737946/11737946 [==============================] - 2404s - loss: 4.5587
Epoch 3/3
11737946/11737946 [==============================] - 2404s - loss: 4.3968

Next, train the sentiment analysis model on 1,000 samples. Since the earlier pretraining has already been done, there aren't many parameters left to train here, and combined with dropout, 1,000 samples won't cause serious overfitting.

#Training the sentiment analysis model

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=200)

The training results were:

Epoch 198/200
1000/1000 [==============================] - 0s - loss: 0.2481 - acc: 0.9250
Epoch 199/200
1000/1000 [==============================] - 0s - loss: 0.2376 - acc: 0.9330
Epoch 200/200
1000/1000 [==============================] - 0s - loss: 0.2386 - acc: 0.9350

Now let's evaluate the model:

#评估一下模型的效果
model.evaluate(x_test, y_test, verbose=True, batch_size=batch_size)

Accuracy: 73.04% — not great. Let's try transfer learning, training on the training set together with the predicted labels on the test set:

#把训练集连同测试集的预测结果（即可能包含有误的数据），重新训练模型
y_pred = model.predict(x_test, verbose=True, batch_size=batch_size)
y_pred = (y_pred.reshape(-1) > 0.5).astype(int)
xt = np.vstack([x_train, x_test])
yt = np.hstack([y_train, y_pred])

model.fit(xt, yt,
          batch_size=batch_size,
          epochs=10)

#评估一下模型的效果
model.evaluate(x_test, y_test, verbose=True, batch_size=batch_size)

The training results were:

Epoch 8/10
50000/50000 [==============================] - 27s - loss: 0.1455 - acc: 0.9561
Epoch 9/10
50000/50000 [==============================] - 27s - loss: 0.1390 - acc: 0.9590
Epoch 10/10
50000/50000 [==============================] - 27s - loss: 0.1349 - acc: 0.9600

This time we got an accuracy of 73.33%. It's not hard to see that this process could in principle be repeated iteratively. Doing it again gives 73.33% accuracy; a second repetition gives 73.47%... One would expect this to converge to a stable value — after repeating it 5 times, it stabilized at 73.48%.

Going from the initial 73.04% to 73.48% after transfer learning is roughly a 0.44% improvement. That may not look like much, but for anyone doing a competition or writing a paper, a 0.44% improvement is definitely worth a mention.

Remarks

As mentioned at the start, this was not a particularly successful attempt — after all, it's a "budget version," so don't get too hung up on the accuracy not being especially high. Still, based on these experimental results, this approach seems sound. Training a language model on a large volume of mixed-sentiment corpus does indeed extract useful text features quite well, which is analogous to the autoencoding process in images.

What I did here was quite simple, without fine-tuning hyperparameters in any detail. There are several possible directions for improvement: scaling up the language model, adding more sentiment-related corpus (it only needs to be sentiment-related text, no labels required), and optimizing the training details. I'll leave those for another time.