Text Sentiment Classification (I): Traditional Models

文本情感分类（一）：传统模型 苏剑林 · 2015-06-22 · Machine Learning Python machine learning data mining text mining

Preface: Back in April and May, I took part in two data-mining-related competitions: the "Liangjian Cup" hosted by the School of Physics and Electronics, and the Third "Teddy Cup" National College Student Data Mining Competition. As it happens, both competitions included a question mainly concerned with Chinese sentiment classification. When working on the Liangjian Cup, since I was still just starting out and my level was limited, I only implemented a simple text sentiment classification model based on a traditional approach. Later, in the Teddy Cup, having studied further, I had a basic grasp of deep learning ideas, and implemented a text sentiment classification model using deep learning algorithms. So I've decided to put both models on the blog for readers' reference. Readers who are just starting out can compare the two approaches and get a feel for the underlying ideas here. Experts, please feel free to smile and move on.

Sentiment-Lexicon-Based Approach

The simplest form of human judgmentmore

The traditional sentiment-lexicon-based approach to text sentiment classification is the simplest simulation of human memory and judgment, as shown in the figure above. First, through learning, we memorize some basic vocabulary — for instance, negation words like "not," positive words like "like" and "love," negative words like "dislike" and "hate," and so on — thereby forming a basic corpus in our minds. Then, we split the input sentence in the most direct way and check whether the words we've memorized appear in it, judging the sentiment based on the category of the matched words. For example, in "I like math," the word "like" is in our memorized list of positive words, so we judge the sentence to carry positive sentiment.

Based on this idea, we can implement sentiment-lexicon-based text sentiment classification through the following steps: preprocessing, tokenization, training the sentiment lexicon, and judgment. The whole process is illustrated in the figure below. The raw material used to test the model includes comments on Mengniu milk provided by Professor Xue Yun, as well as comments on a certain mobile phone model purchased online (see attachment).

Sentiment-lexicon-based text sentiment classification

Text preprocessing

Raw corpora crawled by web crawlers and similar tools usually contain information we don't need, such as extraneous HTML tags, so the corpus needs to be preprocessed. The Mengniu milk comments provided by Professor Xue Yun were no exception. Our team used Python as our preprocessing tool, relying on the NumPy and Pandas libraries, with regular expressions as the main text-processing tool. After preprocessing, the raw corpus was normalized into the table below, where we labeled negative-sentiment comments as -1 and positive-sentiment comments as 1.

$$\begin{array}{c|c|c} \hline & comment & mark\\ \hline 0 & 蒙牛又出来丢人了 & -1\\ 1 & 珍爱生命远离蒙牛 & -1\\ \vdots & \vdots & \vdots \\ 1171 & 我一直都很爱喝蒙牛的纯牛奶 一直，很爱 & 1\\ 1172 & 送蒙牛...健康才是最好的礼物。 & 1\\ \vdots & \vdots & \vdots \\ \hline \end{array}$$

Automatic sentence tokenization

In order to determine whether a sentence contains words that match those in the sentiment lexicon, we need to accurately split the sentence into individual words — that is, perform automatic sentence tokenization. We compared existing tokenization tools, weighing both tokenization accuracy and ease of use on the Python platform, and ultimately chose "Jieba Chinese Tokenizer" as our tokenization tool.

The table below shows how several common tokenization tools handle one representative test sentence:

Test sentence:
工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作

Loading the sentiment lexicon

Generally speaking, the lexicon is the core component of text mining, and text sentiment classification is no exception. Our sentiment lexicon is divided into four parts: a positive-sentiment lexicon, a negative-sentiment lexicon, a negation-word lexicon, and a degree-adverb lexicon. To obtain as complete a sentiment lexicon as possible, we collected several sentiment lexicons available online, merged and deduplicated them, and adjusted some entries in order to achieve as high an accuracy as possible.

Building the sentiment lexicon

Our team didn't simply merge the lexicons collected from the internet — we also cleaned and updated the lexicon in a targeted, purposeful way. In particular, we added certain industry-specific vocabulary to increase the hit rate in classification. The frequency of certain words can vary considerably across industries, and such words may well be key indicators for sentiment classification. For example, the comment data provided by Professor Xue Yun concerned Mengniu milk, i.e., the food and beverage industry; in this industry, the words "eat" and "drink" occur with quite high frequency, and are usually associated with positive evaluations of food, whereas "won't eat" or "won't drink" usually signal negative evaluations — while in other industries or domains these words carry no clear sentiment at all. Another example comes from the mobile phone industry: in a sentence like "This phone is really drop-resistant, and it's waterproof too," "drop-resistant" and "waterproof" are words that carry positive sentiment specifically within the phone domain. It is therefore necessary to factor such considerations into the model.

Text sentiment classification

The classification rule for the sentiment-lexicon-based approach is fairly mechanical. For simplicity, we assign a weight of 1 to every positive-sentiment word and a weight of -1 to every negative-sentiment word, and assume that sentiment values combine linearly. We then tokenize the sentence, and whenever the resulting word vector contains a matching word, we add its corresponding weight — with special handling for negation words and degree adverbs: a negation word flips the sign of the weight, while a degree adverb doubles it. Finally, the sign of the total weight determines the sentiment of the sentence. The basic algorithm is shown in the figure below.

Sentiment-lexicon-based text classification – flowchart

It should be noted that, for the sake of programming and testing feasibility, we made several simplifying assumptions. Assumption 1: we assumed that all positive words carry equal weight, and likewise for all negative words — this only holds under a crude judgment scheme, and clearly fails for more precise classification, since "hate" is obviously stronger than "dislike." A fix for this shortcoming is to assign a different weight to each word; we'll explore ways of assigning such weights in Part 2 of this post. Assumption 2: we assumed the weights combine linearly, which holds in most cases; in Part 2 we'll discuss introducing non-linearity to improve accuracy. Assumption 3: for negation words and degree adverbs, we simply applied a plain sign flip and doubling respectively, but in fact different negation words and degree adverbs carry different weights — for instance, "like very much" is clearly stronger than "quite like," yet we made no distinction between them.

For the implementation, we chose Python as our platform. As you can see, thanks to Python's rich ecosystem of libraries, we implemented all the steps above — obtaining a working sentiment classification algorithm — in fewer than a hundred lines of code, which really shows off Python's conciseness. Below we test the effectiveness of our algorithm.

Model Result Verification

As the most basic test, we first applied our model to the Mengniu milk comments provided by Professor Xue Yun, and the results were satisfying, reaching an accuracy of 82.02%. The detailed evaluation report is shown below:

$$\begin{array}{c|c|c|c|c|c} \hline 数据内容 & 正样本数 & 负样本数 & 准确率 & 真正率 & 真负率\\ \hline 牛奶评论 & 1005 & 1170 & 0.8202 & 0.8209 & 0.8197\\ \hline \end{array}$$

(Here, positive samples are comments with positive sentiment, and negative samples are comments with negative sentiment,

$$\begin{aligned} &\text{accuracy}=\frac{\text{correctly classified samples}}{\text{total sample count}}\\ &\text{true positive rate}=\frac{\text{positive samples predicted positive}}{\text{total positive samples}}\\ &\text{true negative rate}=\frac{\text{number of negative samples judged as negative}}{\text{total negative samples}} \end{aligned}$$.)

To our pleasant surprise, when the model tuned on the Mengniu milk comment data was applied directly to sentiment classification on comment data for a certain mobile phone model, it also achieved 81.96% accuracy! This shows that our model has fairly good robustness, performing well on sentiment classification across comment data from different industries.

$$\begin{array}{c|c|c|c|c|c} \hline 数据内容 & 正样本数 & 负样本数 & 准确率 & 真正率 & 真负率\\ \hline 手机评论 & 1158 & 1159 & 0.8196 & 0.7539 & 0.8852\\ \hline \end{array}$$

Conclusion: Our team implemented a preliminary version of sentiment-lexicon-based text sentiment classification, and the test results show that even simple decision rules can give this algorithm decent accuracy, along with good robustness. It's generally considered that a model with accuracy above 80% has some production value and can be applied in an industrial setting. Clearly, our model has already reached this bar in preliminary terms.

Where the Difficulty Lies

After two rounds of testing, we can tentatively say our model's accuracy is basically above 80%. Meanwhile, some fairly mature commercial systems (such as BosonNLP) only achieve accuracy around 85% to 90%. This shows that our simple model has indeed achieved a satisfying result; on the other hand, it also shows that the traditional "sentiment-lexicon-based text sentiment classification" model has quite limited room for further improvement. This is due to the inherent complexity of text sentiment classification itself. After some initial discussion, we believe the difficulty of text sentiment classification lies in the following aspects.

The language system is remarkably complex

Ultimately, this is because the language system in our brains is remarkably complex. (1) What we're doing here is text sentiment classification, and both text and textual sentiment are products of human culture — in other words, humans are the only accurate criterion for judgment. (2) Human language is a highly complex cultural product; a sentence is not simply a linear combination of words, but contains fairly complex non-linearity. (3) When we describe a sentence, we treat it as a whole rather than as a mere collection of words — different combinations, orders, and numbers of words can all yield different meanings and sentiments, which is precisely what makes text sentiment classification difficult.

Therefore, text sentiment classification work is in essence a simulation of human thought processes. The model described above is, in fact, already the simplest such simulation. However, what we're simulating there is merely some simple fixed patterns of thought; genuine sentiment judgment isn't a matter of a few simple rules, but rather a complex network.

The brain isn't only doing sentiment classification

In fact, when we judge the sentiment of a sentence, we're not only thinking about what sentiment the sentence carries — we're also judging the type of sentence (imperative, interrogative, or declarative?); when considering each word in the sentence, we don't just focus on positive words, negative words, negations, or degree adverbs — we pay attention to every word (subject, predicate, object, etc.), thereby forming an understanding of the sentence as a whole; we may even bring in context to judge the sentence. We may be doing these things unconsciously, but our brains really are doing them, in order to form a complete understanding of the sentence and thereby make an accurate judgment of its sentiment. In other words, our brains are in fact an extremely fast and complex processor — while doing sentiment classification, we're simultaneously doing a great many other things as well.

Living water: learning to predict

What distinguishes humans from machines — and indeed from other animals — is our capacity for awareness and ability to learn. We acquire new knowledge not only by being taught by others, but also through our own learning, summarizing, and guessing. Text sentiment classification is no exception: we can not only memorize large numbers of sentiment words, but also summarize or infer new sentiment words. For instance, if we only know that "like" and "love" both carry positive sentiment, we might guess that "adore" also carries positive sentiment. This capacity for learning is an important way we expand our vocabulary, and it also optimizes our memory (that is, we don't need to specifically cram the word "adore" into our brain's corpus — we only need to remember "like" and "love," and endow them with some connection, in order to arrive at "adore"; this is an optimized mode of memory).

Ideas for Improvement

Based on the analysis above, we've identified the essential complexity of text sentiment classification, as well as several characteristics of how the human brain performs classification. In light of this analysis, we propose the following improvements.

Introducing non-linear features

As mentioned earlier, real human sentiment classification in the brain is in fact heavily non-linear, and models based on simple linear combination have limited performance. So, to improve the model's accuracy, it's necessary to introduce non-linearity into the model.

By non-linearity, we mean that combinations of words form new meanings. In fact, our preliminary model already introduces a simple form of non-linearity — in the model above, we treat adjacent occurrences of a positive word and a negative word as forming a combined negative chunk, and assign it a negative weight. Finer-grained combination weights can be implemented via a "lexicon matrix": we put all known positive and negative words into one set, number them one by one, and then use the following "lexicon matrix" to record the weight of each word pair.

$$\begin{array}{c|cccccc} 词语 & （空词） & 喜欢 & 爱 & \dots & 讨厌 & \dots\\ \hline （空词） & 0 & 1 & 2 & \dots & -1 & \dots\\ 喜欢 & 1 & 2 & 3 & \dots & -2 & \dots\\ 爱 & 2 & 3 & 4 & \dots & -2 & \dots\\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \dots\\ 讨厌 & -1 & -2 & -3 & \dots & -2 & \dots\\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \dots \end{array}$$

Not every combination of words is meaningful, but we can still compute a combination weight for each pair; the calculation of sentiment weights can be found in the reference literature. However, the number of sentiment words is quite large, and the number of entries in the lexicon matrix is its square, so the data volume involved is considerable — this already falls, in a preliminary sense, within the realm of big data. To implement non-linearity more efficiently, we need to explore optimized schemes for constructing word combinations, as well as for storing and indexing them.

Automatic expansion of the sentiment lexicon

In today's networked information age, new words keep springing up like bamboo shoots after rain — including both "newly coined internet words" and "existing words given new meanings." On top of that, the sentiment lexicon we've compiled can never fully cover all existing sentiment words. Therefore, automatically expanding the sentiment lexicon is a necessary condition for keeping the sentiment classification model up to date. Currently, through web crawlers and similar means, we can collect large volumes of comment data from Weibo, forums, and the like; in order to find new sentiment-bearing words within this large batch of data, our approach is unsupervised learning via word-frequency statistics.

Our goal is "automatic expansion," so what we aim to achieve is unsupervised learning based on the existing preliminary model, to complete the lexicon expansion and thereby strengthen the model's own performance — and then iterate the same process again, forming a positive feedback loop. Although we can scrape large amounts of comment data from the internet, this data is unlabeled; we need to use the existing model to classify the sentiment of the comment data, then, within the set of comments sharing the same sentiment (positive or negative), tabulate the frequency of each word, and finally compare the word frequencies between the positive and negative comment sets. If a word's frequency is quite low in the positive comment set but quite high in the negative comment set, then we can be fairly confident in adding that word to the negative sentiment lexicon — in other words, assigning it a negative weight.

For example, suppose our negative sentiment lexicon doesn't yet contain the word "unscrupulous" (黑心), but basic sentiment words like "detestable," "dislike," "disgusted," and "like" are already present in the sentiment lexicon. Then we would be able to correctly classify the sentiment of the following sentences:

$$\begin{array}{c|c} \hline 句子 & 权值\\ \hline 这个黑心老板太可恶了 & -2\\ \hline 我很反感这黑心企业的做法 & -2\\ \hline 很讨厌这家黑心店铺 & -2\\ \hline 这家店铺真黑心！ & 0\\ \hline \vdots & \vdots\\ \hline \end{array}$$

Here, since the negative sentiment lexicon doesn't contain the word "unscrupulous," the sentence "This shop is truly unscrupulous!" would only be classified as neutral (i.e., weight 0). After classification, we tally word frequencies separately among the comments classified as positive and negative, and we find that the new word "unscrupulous" appears many times among negative comments but almost never among positive ones. So we add "unscrupulous" to our negative sentiment lexicon, and then update our classification results:

$$\begin{array}{c|c} \hline 句子 & 权值\\ \hline 这个黑心老板太可恶了 & -3\\ \hline 我很反感这黑心企业的做法 & -3\\ \hline 很讨厌这家黑心店铺 & -3\\ \hline 这家店铺真黑心！ & -2\\ \hline \vdots & \vdots\\ \hline \end{array}$$

In this way, we expand the lexicon through unsupervised learning, while simultaneously improving accuracy and strengthening the model's performance. This is an iterative process, where the result of each step helps drive the next.

Conclusions of this post

Bringing together the analysis above, we arrive at the following conclusions:

Sentiment-lexicon-based text sentiment classification is easy to implement, and its core lies in training the sentiment lexicon.
The language system is remarkably complex, and sentiment-lexicon-based text sentiment classification is only a linear model, so its performance is inherently limited.
Properly introducing non-linear features into text sentiment classification can effectively improve the model's accuracy.
Introducing an unsupervised learning mechanism for lexicon expansion can effectively discover new sentiment words, ensuring the model's robustness and up-to-dateness.

References

Notes on studying deep learning: http://blog.csdn.net/zouxy09/article/details/8775360

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model, 2003

A new language model: http://blog.sciencenet.cn/blog-795431-647334.html

Sentiment analysis dataset of comment data: http://www.datatang.com/data/11857

"Jieba" Chinese Tokenizer: https://github.com/fxsjy/jieba

NLPIR Chinese Word Segmentation System: http://ictclas.nlpir.org/

smallseg: https://code.google.com/p/smallseg/

Yaha Tokenizer: https://github.com/jannson/yaha

Sentiment Analysis Word Set (beta): http://www.keenage.com/html/c_bulletin_2007.htm

NTUSD Simplified Chinese Sentiment Polarity Lexicon: http://www.datatang.com/data/11837

Degree Adverb Intensity and Negation Word List: http://www.datatang.com/data/44198

Compiled Existing Sentiment Lexicons: http://www.datatang.com/data/46922

BosonNLP: http://bosonnlp.com/product

Implementation Platform

Our team's programming tools were tested in the following environment:

Windows 8.1 Microsoft operating system.
Python 3.4, development platform/programming language. The main reason for choosing 3.x over 2.x was that 3.x offers better support for Chinese characters.
NumPy A Python numerical computing library, providing Python with fast multi-dimensional array processing capability.
Pandas A Python data analysis package.
Jieba Tokenizer A Chinese tokenization tool for the Python platform, also available in Java, C++, Node.js, and other versions.

Code Listings

Resource: Sentiment Polarity Lexicon.zip

Preprocessing

#-*- coding: utf-8 -*-

import numpy as np #导入numpy
import pandas as pd
import jieba

def yuchuli(s,m): #导入文本，文本预处理
	wenjian = pd.read_csv(s, delimiter='     xxx     ', encoding='utf-8', \
	header= None, names=['comment']) #导入文本
	wenjian = wenjian['comment'].str.replace('(<.*?>.*?<.*?>)','').str.replace('(<.*?>)','')\
	.str.replace('(@.*?[ :])',' ') #替换无用字符
	wenjian = pd.DataFrame({'comment':wenjian[wenjian != '' ]})
	wenjian.to_csv('out_'+s, header=False, index=False)
	wenjian['mark'] = m #样本标记
	return wenjian.reset_index()

neg = yuchuli('data_neg.txt',-1)
pos = yuchuli('data_pos.txt',1)

mydata = pd.concat([neg,pos],ignore_index=True)[['comment','mark']] #结果文件
#预处理基本结束

Loading the sentiment lexicon

#开始加载情感词典
negdict = [] #消极情感词典
posdict = [] #积极情感词典
nodict = [] #否定词词典
plusdict = [] #程度副词词典
sl = pd.read_csv('dict/neg.txt', header=None, encoding='utf-8')
for i in range(len(sl[0])):
	negdict.append(sl[0][i])
sl = pd.read_csv('dict/pos.txt', header=None, encoding='utf-8')
for i in range(len(sl[0])):
	posdict.append(sl[0][i])
sl = pd.read_csv('dict/no.txt', header=None, encoding='utf-8')
for i in range(len(sl[0])):
	nodict.append(sl[0][i])
sl = pd.read_csv('dict/plus.txt', header=None, encoding='utf-8')
for i in range(len(sl[0])):
	plusdict.append(sl[0][i])
#加载情感词典结束

Prediction function

#预测函数
def predict(s, negdict, posdict, nodict, plusdict):
	p = 0
	sd = list(jieba.cut(s))
	for i in range(len(sd)):
		if sd[i] in negdict:
			if i>0 and sd[i-1] in nodict:
				p = p + 1
			elif i>0 and sd[i-1] in plusdict:
				p = p - 2
			else: p = p - 1
		elif sd[i] in posdict:
			if i>0 and sd[i-1] in nodict:
				p = p - 1
			elif i>0 and sd[i-1] in plusdict:
				p = p + 2
			elif i>0 and sd[i-1] in negdict:
				p = p - 1
			elif i<len(sd)-1 and sd[i+1] in negdict:
				p = p - 1
			else: p = p + 1
		elif sd[i] in nodict:
			p = p - 0.5
	return p
#预测函数结束

A simple test

#简单的测试
tol = 0
yes = 0
mydata['result'] = 0
for i in range(len(mydata)):
	print(i)
	tol = tol + 1
	if predict(mydata.loc[i,'comment'], negdict, posdict, nodict, plusdict)*mydata.loc[i,'mark'] > 0:
		yes = yes + 1
		mydata.loc[i,'result'] = 1
		
print(yes/tol)