Text Sentiment Classification (II): Deep Learning Models

文本情感分类（二）：深度学习模型 苏剑林 · 2015-08-04 · Machine Learning Python deep learning machine learning text mining

Language Processing

In Text Sentiment Classification (I): Traditional Models, I gave a brief introduction to the traditional approach to text sentiment classification. The traditional approach is simple and reasonably stable, but it has two hard-to-overcome limitations: first, an accuracy problem — the traditional approach delivers only so-so results; it's good enough for general applications, but there's no good way to push the accuracy further. Second, a background-knowledge problem — the traditional approach requires a sentiment lexicon to be prepared in advance, and that step usually needs manual work to guarantee accuracy. In other words, the person doing this work needs to be not just a data-mining expert but also a linguist. This dependency on background knowledge stands in the way of progress in natural language processing.more

Fortunately, deep learning solves this problem (or at least solves it to a large extent): it lets us build a model for a real-world problem in some domain with almost "zero background knowledge." This post continues with the text sentiment classification example from the previous post and gives a brief explanation of the deep learning model. Parts that were already discussed in detail in the previous post won't be elaborated on again here.

Deep Learning and Natural Language Processing

In recent years, deep learning algorithms have been applied to natural language processing and have achieved better results than traditional models. For example, Bengio and colleagues built a neural probabilistic language model based on deep learning ideas, and further used various deep neural networks to train language models on large-scale English corpora, obtaining good semantic representations and completing common NLP tasks such as syntactic parsing and sentiment classification — offering new ideas for natural language processing in the big-data era.

Based on my own testing, deep-neural-network-based sentiment analysis models often achieve accuracy above 95% — a good demonstration of the appeal and power of deep learning algorithms!

For further reading on deep learning, please refer to the following references:

[1] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model, 2003
[2] A New Language Model: http://blog.sciencenet.cn/blog-795431-647334.html
[3] Notes on Deep Learning: http://blog.csdn.net/zouxy09/article/details/8775360
[4] Deep Learning: http://deeplearning.net
[5] A Casual Discussion on Chinese Word Segmentation and Semantic Recognition: http://www.matrix67.com/blog/archives/4212
[6] Applications of Deep Learning in Chinese Word Segmentation and Part-of-Speech Tagging: http://blog.csdn.net/itplus/article/details/13616045

Representing Language

In the post Idle Chat: Neural Networks and Deep Learning, I already mentioned that the most important step in modeling is feature extraction, and natural language processing is no exception. The core question in NLP is: how can we effectively represent a sentence numerically? Once this step is solved, sentence classification is no longer a problem. Obviously, the most naive idea is: assign each word a unique index — 1, 2, 3, 4, ... — and treat a sentence as a collection of indices. For example, suppose 1, 2, 3, 4 stand for "I", "you", "love", "hate" respectively; then "I love you" would be [1, 3, 2], and "I hate you" would be [1, 4, 2]. This idea seems workable, but it's actually quite problematic. For instance, a stable model would consider 3 and 4 to be close to each other, so it would expect [1, 3, 2] and [1, 4, 2] to give similar classification results. But according to our indexing, the words represented by 3 and 4 have completely opposite meanings, so the classification results cannot be the same. Hence, this encoding scheme cannot give good results.

Readers might think: what if I group words with similar meanings together (giving them similar indices)? Indeed, if there were a way to place similar words next to each other in the index, that would greatly improve the model's accuracy. But here's the problem: if we give each word a unique index and arrange similar words to have similar indices, we are effectively assuming that semantics is one-dimensional. But that's not the case — semantics should be multi-dimensional.

For example, when we mention "home," some people will think of the near-synonym "family," and from "family" they might think of "relatives" — these are all words with similar meanings. On the other hand, from "home," some people might think of "Earth," and from "Earth," "Mars." In other words, both "relatives" and "Mars" can be seen as second-order associations of "home," but "relatives" and "Mars" themselves have no obvious connection to each other. Furthermore, semantically speaking, "university" and "comfortable" could also be regarded as second-order associations of "home." Clearly, if we only have a single unique index, it's very hard to place these words in appropriate positions.

Divergence of word meanings

Word2Vec: Here Comes High Dimensionality

From the discussion above, we can see that the meanings of many words fan out in various directions rather than along a single direction, so a single index isn't ideal. So what about multiple indices? In other words, mapping each word to a multi-dimensional vector? Indeed — this is exactly the right idea.

Why do multi-dimensional vectors work? First, they solve the problem of words diverging in multiple directions — even a two-dimensional vector can rotate through a full 360 degrees, let alone higher dimensions (in practice, usually a few hundred dimensions). Second, there's a more practical issue: multi-dimensional vectors let us represent words using numbers with a much smaller range of variation. What do I mean by this? We know that, for Chinese, there are hundreds of thousands of words. If we gave each word a unique index, the index would range from 1 all the way up to several hundred thousand — such a huge range makes it very hard to guarantee the model's stability. With a high-dimensional vector, say 20 dimensions, we only need 0s and 1s to represent $2^{20} = 1048576$ (one million) words. A smaller range of variation helps ensure the model's stability.

I've rambled on for a while without really getting to the point. Now we have an idea, but the question is: how do we actually place these words into the correct high-dimensional vectors? And crucially, how do we do this without any language background at all? (In other words, if I want to handle an English-language task, I don't need to learn English first — I just need to collect a large amount of English text. How convenient would that be!) There's no need — nor is it necessary — for us to go into more theoretical depth here; instead, let's introduce a well-known tool built on exactly this idea and open-sourced by Google — Word2Vec.

Simply put, Word2Vec accomplishes exactly what we described above: representing words with high-dimensional vectors (word vectors, or word embeddings) and placing words with similar meanings in nearby positions, using real-valued vectors (not restricted to integers). All we need is a large corpus of some language, and we can use it to train a model and obtain word vectors. I've already mentioned some of the benefits of word vectors — or rather, they were created precisely to solve the problems mentioned earlier. Other benefits include: word vectors make clustering easy, and Euclidean distance or cosine similarity can be used to find two words with similar meanings. This effectively solves the problem of "multiple words, one meaning" (unfortunately, there doesn't seem to be a good way yet to solve the reverse problem of "one word, multiple meanings.")

For the mathematical principles behind Word2Vec, readers can refer to this series of articles. As for implementation, Google officially provides C source code, which readers can compile themselves. Python's Gensim library also provides a ready-made Word2Vec submodule (in fact, this version seems to be even more powerful than the official one).

Representing Sentences: Sentence Vectors

The next problem to solve is this: once we've segmented the words and converted them into high-dimensional vectors, a sentence corresponds to a collection of word vectors — that is, a matrix, similar to how a digitized image corresponds to a pixel matrix. But models generally only accept one-dimensional feature inputs — so what do we do? A relatively simple idea is to flatten the matrix, that is, to concatenate the word vectors one after another into one long vector. This idea works in principle, but it would push the input dimension up to several thousand or even tens of thousands of dimensions, which is impractical in practice. (If tens of thousands of dimensions isn't a problem for today's computers, then for a 1000×1000 image, that would be as high as one million dimensions!)

In fact, for image processing there's already a mature set of methods called convolutional neural networks (CNNs) — a type of neural network specifically designed to handle matrix-shaped inputs, capable of encoding matrix inputs into lower-dimensional one-dimensional vectors while retaining most of the useful information. The whole CNN toolkit can also be directly transplanted to natural language processing, especially to text sentiment classification, and works reasonably well; a related paper is Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts. But the underlying nature of sentences differs from that of images. Directly applying the image-processing toolkit to language, while achieving some modest success, always feels a bit awkward and out of place. Hence, this is not the mainstream approach in NLP.

In natural language processing, the commonly used approach is the recursive neural network or recurrent neural network (both abbreviated RNNs). Their role is the same as that of CNNs: encoding matrix-form input into a lower-dimensional one-dimensional vector while preserving most of the useful information. The difference from CNNs is that CNNs focus more on global, blurry perception (much like looking at a photo — we don't actually see any single pixel clearly, but rather grasp the picture as a whole), whereas RNNs focus on reconstructing information from neighboring positions. This makes RNNs more convincing for language tasks (language is always built up from adjacent characters forming words, adjacent words forming phrases, adjacent phrases forming sentences, and so on — so we need to effectively integrate, or "reconstruct," information from neighboring positions).

When it comes to classifying models, the variety is practically endless. Within the RNN family alone there are many variants, such as vanilla RNNs, GRU, LSTM, and so on. Readers can refer to Keras's official documentation: http://keras.io/models/ — Keras is a Python deep learning library that provides a large number of deep learning models. Its official documentation serves both as a tutorial and as a catalog of models — it essentially implements most of the currently popular deep learning models.

Building an LSTM Model

Enough chit-chat — time to get something done. Now let's build a deep learning model for text sentiment classification based on LSTM (Long-Short Term Memory). Its structure is shown below:

Using LSTM for sentiment classification

The model structure is quite simple, nothing complicated, and easy to implement — we're using Keras, which has already implemented these algorithms for us out of the box.

Now let's talk about two interesting steps.

The first step is collecting labeled corpora. Note that our model is trained under supervision (or at least semi-supervision), so we need to collect sentences that have already been classified — and naturally, the more the better. For Chinese text sentiment classification, this step is genuinely difficult, since Chinese-language resources are often quite scarce. While building this model, I scraped together, through various channels (some downloaded from online searches, some purchased from Datatang), a corpus of over 20,000 labeled Chinese sentences (spanning six domains) to train the model. (Shared at the end of this post.)

Training corpus

The second step concerns the choice of model threshold. In practice, the prediction result from training is a continuous real number in the interval [0, 1], and by default the program sets 0.5 as the threshold — that is, results above 0.5 are classified as positive and results below 0.5 as negative. This default is often not optimal. As shown in the figure below, while studying the effect of different thresholds on the true positive rate and true negative rate, we found a sharp change in the curves within the interval (0.391, 0.394).

Choice of threshold

Although in absolute terms the value only dropped from 0.99 to 0.97 — not a huge change — the rate of change is quite large. Normally, everything changes smoothly, so a sharp jump like this implies that something unusual is going on, and it's obviously hard to pin down the exact cause. In other words, there exists an unstable region here, within which the predictions are, in fact, unreliable. So, to be safe, we discard this interval: only results above 0.394 are treated as positive, only results below 0.391 are treated as negative, and anything between 0.391 and 0.394 is left undetermined. Experiments show that this practice helps improve the model's real-world accuracy.

A Few Concluding Remarks

This has been a long post, giving a rough overview of the ideas and practical applications of deep learning in text sentiment classification — much of it discussed only in broad strokes. My goal was never to write a full tutorial on deep learning, but simply to point out what I consider to be the key points. There are many excellent tutorials on deep learning out there; it's best to read the English-language papers. As for Chinese-language resources, the best one is probably the blog http://blog.csdn.net/itplus, so I won't embarrass myself by trying to compete on that front.

Below are my corpus and code. You might wonder why I'm sharing these "private treasures." The answer is simple: this isn't my day job — data mining is just a hobby for me, a hobby that combines math and Python. So in this area, I don't need to worry about anyone getting ahead of me.

Corpus download: sentiment.zip

Collected review data: sum.zip

Code for building the LSTM-based text sentiment classification model:

import pandas as pd #导入Pandas
import numpy as np #导入Numpy
import jieba #导入结巴分词

from keras.preprocessing import sequence
from keras.optimizers import SGD, RMSprop, Adagrad
from keras.utils import np_utils
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.embeddings import Embedding
from keras.layers.recurrent import LSTM, GRU

from __future__ import absolute_import #导入3.x的特征函数
from __future__ import print_function

neg=pd.read_excel('neg.xls',header=None,index=None)
pos=pd.read_excel('pos.xls',header=None,index=None) #读取训练语料完毕
pos['mark']=1
neg['mark']=0 #给训练语料贴上标签
pn=pd.concat([pos,neg],ignore_index=True) #合并语料
neglen=len(neg)
poslen=len(pos) #计算语料数目

cw = lambda x: list(jieba.cut(x)) #定义分词函数
pn['words'] = pn[0].apply(cw)

comment = pd.read_excel('sum.xls') #读入评论内容
#comment = pd.read_csv('a.csv', encoding='utf-8')
comment = comment[comment['rateContent'].notnull()] #仅读取非空评论
comment['words'] = comment['rateContent'].apply(cw) #评论分词 

d2v_train = pd.concat([pn['words'], comment['words']], ignore_index = True) 

w = [] #将所有词语整合在一起
for i in d2v_train:
    w.extend(i)

dict = pd.DataFrame(pd.Series(w).value_counts()) #统计词的出现次数
del w,d2v_train
dict['id']=list(range(1,len(dict)+1))

get_sent = lambda x: list(dict['id'][x])
pn['sent'] = pn['words'].apply(get_sent) #速度太慢

maxlen = 50

print("Pad sequences (samples x time)")
pn['sent'] = list(sequence.pad_sequences(pn['sent'], maxlen=maxlen))

x = np.array(list(pn['sent']))[::2] #训练集
y = np.array(list(pn['mark']))[::2]
xt = np.array(list(pn['sent']))[1::2] #测试集
yt = np.array(list(pn['mark']))[1::2]
xa = np.array(list(pn['sent'])) #全集
ya = np.array(list(pn['mark']))

print('Build model...')
model = Sequential()
model.add(Embedding(len(dict)+1, 256))
model.add(LSTM(128)) # try using a GRU instead, for fun
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(x, y, batch_size=16, nb_epoch=10) #训练时间为若干个小时

classes = model.predict_classes(xt)
acc = np_utils.accuracy(classes, yt)
print('Test accuracy:', acc)