Text Sentiment Classification (III): To Tokenize or Not to Tokenize

文本情感分类（三）：分词 OR 不分词 苏剑林 · 2016-06-29 · Machine Learning Python deep learning text mining

After last year's Teddy Cup competition, I wrote a short blog post introducing the application of deep learning to sentiment analysis, Text Sentiment Classification (II): Deep Learning Models]. Although that post was fairly rough, it drew quite a bit of reader feedback, which caught me by surprise. However, some parts of the implementation in that post were unclear, for two reasons: 1) since that post, Keras has undergone fairly significant changes, so the original code no longer works; 2) the code in that post may have been casually modified by me at some point, so the version I posted wasn't quite the right one. So, nearly a year later, I'm picking this topic back up and completing some tests I never got around to finishing before.

Why use deep learning models? Aside from higher accuracy and other such reasons, there's another important one: it is currently the only approach capable of achieving "end-to-end" modeling. By "end-to-end", I mean feeding in the raw data and labels directly, and letting the model handle everything itself—including feature extraction and learning. Looking back at how we typically do Chinese sentiment classification, the process usually goes through several steps: "tokenization → word embedding → sentence embedding (LSTM) → classification." Although this kind of model has often achieved state-of-the-art results, some questions still deserve further testing. For Chinese, the character is the smallest meaningful unit of text, so from an "end-to-end" perspective, we should feed the sentence in directly at the character level, rather than tokenizing it into words first. So is tokenization actually necessary? This post tests and compares the performance of character one-hot encoding, character embeddings, and word embeddings.

Model Tests

This post tests three models—or rather, three frameworks—with the full code given at the end. The three frameworks are:

1. one hot: operating at the character level, without tokenization, truncating each sentence to 200 characters (padding with empty strings if shorter), then feeding the sentence into an LSTM model as a "character-one hot" matrix for classification;
2. one embedding: operating at the character level, without tokenization, truncating each sentence to 200 characters (padding with empty strings if shorter), then feeding the sentence into an LSTM model as a "character-embedding" matrix for classification;
3. word embedding: operating at the word level, with tokenization, truncating each sentence to 100 words (padding with empty strings if shorter), then feeding the sentence into an LSTM model as a "word-embedding" matrix for classification.

The LSTM architecture used in each case is similar. The corpus used is the same one from Text Sentiment Classification (II): Deep Learning Models], with 15,000 samples used for training and the remaining roughly 6,000 for testing. Surprisingly, all three models achieved similar results.

$$\begin{array}{c|ccc} \hline &\text{one hot} & \text{one embedding} & \text{word embedding}\\ \hline \text{num iterations} & 90 & 30 & 30\\ \text{time per epoch} & 100s & 36s & 18s\\ \text{training accuracy} & 96.60\% & 95.95\% & 98.41\% \\ \text{test accuracy} & 89.21\% & 89.55\% & 89.03\% \\ \hline \end{array}$$

As we can see, in terms of accuracy, the three are quite similar, with little to distinguish them. Whether we use one-hot encoding, character embeddings, or word embeddings, the results are roughly the same. Perhaps using the method from Text Sentiment Classification (II): Deep Learning Models] to select an appropriate threshold for each model would push the test accuracy a bit higher, but the relative ranking between the models probably wouldn't change much.

Of course, the tests themselves may not be entirely fair, which could have skewed the results somewhat—and I didn't repeat the experiments extensively to check. For instance, the one-hot model was trained for 90 epochs, while the other two models were trained for only 30, because the samples constructed for the one-hot model have such enormous dimensionality that it takes much longer to converge. Moreover, during training its accuracy rises in a fluctuating manner rather than climbing steadily like the other two models. This, in fact, seems to be a common characteristic of one-hot models in general.

A Few More Thoughts

It seems that the one-hot model does indeed suffer from the curse of dimensionality, and its training time is much longer besides, without any clear improvement in performance. Does that mean there's no point in studying one-hot representations at all?

I don't think so. Back when people criticized one-hot models, the objection wasn't just the curse of dimensionality—there was also the "semantic gap" problem, meaning that any two words have no correlation whatsoever with each other (whether measured by Euclidean distance or cosine similarity, the result is the same for any pair of words). But while this assumption doesn't hold for words, doesn't it make a lot more sense when applied to Chinese characters? There aren't many Chinese characters that form standalone words; most words consist of two characters. In other words, the assumption that any two characters are uncorrelated is approximately true at the level of individual Chinese characters! And since we then use an LSTM—which inherently has the ability to integrate neighboring pieces of information—the model implicitly carries out the process of combining characters into words.

Besides, the one-hot model has another very important property—it loses no information at all. From a one-hot encoding, we can decode exactly which characters or words made up the original sentence; by contrast, I can't recover the original word from a word embedding vector. All of this suggests that, in many situations, one-hot models still have real value.

So why do we use word embeddings at all? A word embedding essentially makes an assumption: each word has a reasonably well-defined meaning. This assumption is approximately true at the level of words, since words with multiple, unrelated meanings are, after all, relatively rare. Precisely because of this, we can place words into a lower-dimensional real-valued space, representing each word as a real-valued vector, and use the distance or cosine similarity between these vectors to represent the similarity between words. This is also why word embeddings are good at capturing "different words, same meaning" but not so good at capturing "one word, multiple meanings."

Seen this way, of the three models above, only one-hot and word-embedding models really make theoretical sense, while the one-embedding model ends up looking a bit awkward, since individual characters don't really have a well-defined meaning of their own. So why does one embedding still perform reasonably well? My guess is that this is because binary classification is a fairly coarse-grained task (0 or 1); if the task involved finer-grained, multi-class classification, the one-embedding approach might well perform worse. That said, I haven't run more extensive tests on this, since it's quite time-consuming.

Of course, this is all just my personal speculation, and I welcome any corrections. The evaluation of the one-embedding approach in particular is certainly open to debate.

Here's the Code

Some of you probably don't want to sit through my rambling and just want to see the code directly—so here it is, for all three models. Ideally you should use GPU acceleration, especially for the one-hot experiment, or you'll be waiting forever.

Model 1: one hot

# -*- coding:utf-8 -*-

'''
one hot测试
在GTX960上，约100s一轮
经过90轮迭代，训练集准确率为96.60%，测试集准确率为89.21%
Dropout不能用太多，否则信息损失太严重
'''

import numpy as np
import pandas as pd

pos = pd.read_excel('pos.xls', header=None)
pos['label'] = 1
neg = pd.read_excel('neg.xls', header=None)
neg['label'] = 0
all_ = pos.append(neg, ignore_index=True)

maxlen = 200 #截断字数
min_count = 20 #出现次数少于该值的字扔掉。这是最简单的降维方法

content = ''.join(all_[0])
abc = pd.Series(list(content)).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(len(abc)))
word_set = set(abc.index)

def doc2num(s, maxlen): 
    s = [i for i in s if i in word_set]
    s = s[:maxlen]
    return list(abc[s])

all_['doc2num'] = all_[0].apply(lambda s: doc2num(s, maxlen))

#手动打乱数据
#当然也可以把这部分加入到生成器中
idx = list(range(len(all_)))
np.random.shuffle(idx)
all_ = all_.loc[idx]

#按keras的输入要求来生成数据
x = np.array(list(all_['doc2num']))
y = np.array(list(all_['label']))
y = y.reshape((-1,1)) #调整标签形状

from keras.utils import np_utils
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.layers import LSTM
import sys
sys.setrecursionlimit(10000) #增大堆栈最大深度(递归深度)，据说默认为1000，报错

#建立模型
model = Sequential()
model.add(LSTM(128, input_shape=(maxlen,len(abc)))) 
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

#单个one hot矩阵的大小是maxlen*len(abc)的，非常消耗内存
#为了方便低内存的PC进行测试，这里使用了生成器的方式来生成one hot矩阵
#仅在调用时才生成one hot矩阵
#可以通过减少batch_size来降低内存使用，但会相应地增加一定的训练时间
batch_size = 128
train_num = 15000

#不足则补全0行
gen_matrix = lambda z: np.vstack((np_utils.to_categorical(z, len(abc)), np.zeros((maxlen-len(z), len(abc)))))

def data_generator(data, labels, batch_size): 
    batches = [list(range(batch_size*i, min(len(data), batch_size*(i+1)))) for i in range(len(data)/batch_size+1)]
    while True:
        for i in batches:
            xx = np.zeros((maxlen, len(abc)))
            xx, yy = np.array(map(gen_matrix, data[i])), labels[i]
            yield (xx, yy)

model.fit_generator(data_generator(x[:train_num], y[:train_num], batch_size), samples_per_epoch=train_num, nb_epoch=30)

model.evaluate_generator(data_generator(x[train_num:], y[train_num:], batch_size), val_samples=len(x[train_num:]))

def predict_one(s): #单个句子的预测函数
    s = gen_matrix(doc2num(s, maxlen))
    s = s.reshape((1, s.shape[0], s.shape[1]))
    return model.predict_classes(s, verbose=0)[0][0]

Model 2: one embedding

# -*- coding:utf-8 -*-

'''
one embedding测试
在GTX960上，36s一轮
经过30轮迭代，训练集准确率为95.95%，测试集准确率为89.55%
Dropout不能用太多，否则信息损失太严重
'''

import numpy as np
import pandas as pd

pos = pd.read_excel('pos.xls', header=None)
pos['label'] = 1
neg = pd.read_excel('neg.xls', header=None)
neg['label'] = 0
all_ = pos.append(neg, ignore_index=True)

maxlen = 200 #截断字数
min_count = 20 #出现次数少于该值的字扔掉。这是最简单的降维方法

content = ''.join(all_[0])
abc = pd.Series(list(content)).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(1, len(abc)+1))
abc[''] = 0 #添加空字符串用来补全
word_set = set(abc.index)

def doc2num(s, maxlen): 
    s = [i for i in s if i in word_set]
    s = s[:maxlen] + ['']*max(0, maxlen-len(s))
    return list(abc[s])

all_['doc2num'] = all_[0].apply(lambda s: doc2num(s, maxlen))

#手动打乱数据
idx = list(range(len(all_)))
np.random.shuffle(idx)
all_ = all_.loc[idx]

#按keras的输入要求来生成数据
x = np.array(list(all_['doc2num']))
y = np.array(list(all_['label']))
y = y.reshape((-1,1)) #调整标签形状

from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Embedding
from keras.layers import LSTM

#建立模型
model = Sequential()
model.add(Embedding(len(abc), 256, input_length=maxlen))
model.add(LSTM(128)) 
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

batch_size = 128
train_num = 15000

model.fit(x[:train_num], y[:train_num], batch_size = batch_size, nb_epoch=30)

model.evaluate(x[train_num:], y[train_num:], batch_size = batch_size)

def predict_one(s): #单个句子的预测函数
    s = np.array(doc2num(s, maxlen))
    s = s.reshape((1, s.shape[0]))
    return model.predict_classes(s, verbose=0)[0][0]

Model 3: word embedding

# -*- coding:utf-8 -*-

'''
word embedding测试
在GTX960上，18s一轮
经过30轮迭代，训练集准确率为98.41%，测试集准确率为89.03%
Dropout不能用太多，否则信息损失太严重
'''

import numpy as np
import pandas as pd
import jieba

pos = pd.read_excel('pos.xls', header=None)
pos['label'] = 1
neg = pd.read_excel('neg.xls', header=None)
neg['label'] = 0
all_ = pos.append(neg, ignore_index=True)
all_['words'] = all_[0].apply(lambda s: list(jieba.cut(s))) #调用结巴分词

maxlen = 100 #截断词数
min_count = 5 #出现次数少于该值的词扔掉。这是最简单的降维方法

content = []
for i in all_['words']:
	content.extend(i)

abc = pd.Series(content).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(1, len(abc)+1))
abc[''] = 0 #添加空字符串用来补全
word_set = set(abc.index)

def doc2num(s, maxlen): 
    s = [i for i in s if i in word_set]
    s = s[:maxlen] + ['']*max(0, maxlen-len(s))
    return list(abc[s])

all_['doc2num'] = all_['words'].apply(lambda s: doc2num(s, maxlen))

#手动打乱数据
idx = list(range(len(all_)))
np.random.shuffle(idx)
all_ = all_.loc[idx]

#按keras的输入要求来生成数据
x = np.array(list(all_['doc2num']))
y = np.array(list(all_['label']))
y = y.reshape((-1,1)) #调整标签形状

from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Embedding
from keras.layers import LSTM

#建立模型
model = Sequential()
model.add(Embedding(len(abc), 256, input_length=maxlen))
model.add(LSTM(128)) 
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

batch_size = 128
train_num = 15000

model.fit(x[:train_num], y[:train_num], batch_size = batch_size, nb_epoch=30)

model.evaluate(x[train_num:], y[train_num:], batch_size = batch_size)

def predict_one(s): #单个句子的预测函数
    s = np.array(doc2num(list(jieba.cut(s)), maxlen))
    s = s.reshape((1, s.shape[0]))
    return model.predict_classes(s, verbose=0)[0][0]