A More Elegant Word Embedding Model (V): Some Interesting Results
Finally, let's look at what nice properties the word embedding model $(15)$ might have—or in other words, after going to all this trouble to construct a new word embedding model, what do we actually get in return?
What the Norm Means
It seems that almost no word embedding model pays much attention to the norm (length) of the word vectors. Interestingly, the word vectors produced by our model above have norms that, to some extent, reflect how "important" a word is. We can understand this fact from two angles.
Within a context window, the chance of the center word repeating itself is actually fairly small—it's a fairly random event—so we can roughly assume
\[P(w,w) \sim P(w)\tag{24}\]
Then, according to our model, we get
\[e^{\langle\boldsymbol{v}_{w},\boldsymbol{v}_{w}\rangle} =\frac{P(w,w)}{P(w)P(w)}\sim \frac{1}{P(w)}\tag{25}\]
and therefore
\[\Vert\boldsymbol{v}_{w}\Vert^2 \sim -\log P(w)\tag{26}\]
We can see that the higher the frequency of a word (i.e., the more likely it is to be a stopword or function word), the smaller its corresponding word vector norm. This confirms that the norm of this kind of word vector really does capture the word's importance. In fact, the quantity $-\log P(w)$ is similar to IDF; it has a specific name, ICF, discussed in the paper TF-ICF: A New Term Weighting Scheme for Clustering Dynamic Data Streams. more
We can also understand this from another angle. First, decompose each vector into its norm and direction:
\[\boldsymbol{v}=\Vert\boldsymbol{v}\Vert\cdot\frac{\boldsymbol{v}}{\Vert\boldsymbol{v}\Vert}\tag{27}\]
where $|\boldsymbol{v}|$, the norm, is a single independent parameter, and the direction vector $\boldsymbol{v}/\Vert\boldsymbol{v}\Vert$ has $n-1$ independent parameters, with $n$ being the dimensionality of the word vectors. Because the number of parameters involved differs so much, when solving for the word vectors, if something can be achieved simply by adjusting the norm, the model will naturally prefer to adjust the norm rather than laboriously adjust the direction. And according to $(15)$, we have
\[\log\frac{P(w_i,w_j)}{P(w_i)P(w_j)}=\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle=\Vert\boldsymbol{v}_i\Vert\cdot \Vert\boldsymbol{v}_i\Vert\cdot \cos\theta_{ij}\tag{28}\]
For nearly meaningless words like "的" ("of/'s") or "了" (a grammatical particle), which direction will their word vectors move in? As mentioned earlier, these words occur very frequently, yet they are barely fixed collocations with any particular word—they basically just wander around among themselves. So we can assume that for any word $w_i$,
\[\log\frac{P(w_i,\text{of})}{P(w_i)P(\text{of})}\approx 0\tag{29}\]
The most convenient way to achieve this is naturally $\Vert\boldsymbol{v}_{\text{of}}\Vert\approx 0$—adjusting a single parameter is enough, and the model is certainly happy to do that. In other words, for words with high frequency but overall low mutual information with everything else (these words typically carry no special meaning), the norm will automatically shrink toward 0. This is why we say the norm of a word vector can, to some extent, represent how important the word is.
In a set of word vectors trained with the model in this post on the Baidu Baike corpus, without truncating the weights, if we sort the word vectors in ascending order of norm, the first 50 results are:
\[\begin{array}{|c|c|c|c|c|c|c|c|c|c|} \hline \text{。} & \text{，} & \text{of} & \text{and} & \text{similarly} & \text{also} & \text{1} & \text{3} & \text{and} & \text{additionally} \\ \hline \text{meanwhile} & \text{is} & \text{2} & \text{6} & \text{in short} & \text{in} & \text{and} & \text{5} & \text{thus} & \text{4} \\ \hline \text{7} & \text{8} & \text{etc} & \text{also} & \text{and} & \text{；} & \text{meanwhile} & \text{however} & \text{among} & \text{in fact}\\ \hline \text{obviously} & \text{thus} & \text{so} & \text{e.g.} & \text{still} & \text{of course} & \text{is} & \text{these} & \text{and} & \text{hence} \\ \hline \text{besides} & \text{）} & \text{is} & \text{even if} & \text{for example} & \text{since} & \text{thus it can be seen} & \text{one} & \text{has} & \text{i.e.} \\ \hline \end{array}\]
Indeed, these are words we would call "stopwords" or "function words," which confirms that the norm really does reflect a word's own importance. This result is somewhat sensitive to whether the weights are truncated, because with weight truncation, the resulting ranking is:
\[\begin{array}{|c|c|c|c|c|c|c|c|c|c|} \hline \text{。} & \text{，} & \text{in short} & \text{similarly} & \text{meanwhile} & \text{besides} & \text{among} & \text{is} & \text{obviously} & \text{whether} \\ \hline \text{additionally} & \text{not only} & \text{in fact} & \text{thus it can be seen} & \text{even if} & \text{originally} & \text{first} & \text{next} & \text{the latter} & \text{originally} \\ \hline \text{originally} & \text{initially} & \text{for this} & \text{another} & \text{second} & \text{worth mentioning} & \text{observe} & \text{initially} & \text{or} & \text{basically} \\ \hline \text{another} & \text{previously} & \text{as} & \text{since} & \text{denote} & \text{such as} & \text{nowadays} & \text{then} & \text{yet is} & \text{if we say} \\ \hline \text{thus} & \text{indeed} & \text{on the other hand} & \text{after that} & \text{besides} & \text{included} & \text{of course} & \text{the former} & \text{reason} & \text{besides} \\ \hline \end{array}\]
The obvious difference between the two tables is that in the second one, although the words are still mostly stopwords, some of the more obvious stopwords—like "的" ("of/'s") and "是" ("is")—are no longer at the top. This is because their word frequencies are quite large, so truncation has a bigger effect on them, meaning there is a possibility of underfitting (put simply, more attention is paid to low-frequency words, while high-frequency words only need to be "roughly right"). So why do the period and comma, which are also very high-frequency, still make the list? Because within a single window of a sentence, the probability of the period "。" occurring twice is much smaller than the probability of "的" occurring twice, so the period "。" fits our assumption above much better. Correspondingly, since a window may contain multiple occurrences of "的", its mutual information with itself should be larger, so its norm ends up being larger too.
Word Analogy Experiments
Since we've claimed that the word analogy property is essentially the defining feature of this model, does the model actually perform well at word analogies? Let's look at some examples.
\[\begin{array}{c|c} \hline A + B - C& D \\ \hline 机场 + 火车 - 飞机 & 火车站、直达、东站、高铁站、南站、客运站 \\ \hline 国王 + 女人 - 男人& 二世、一世、王后、王国、三世、四世\\ \hline 北京 + 英国 - 中国& 伦敦、巴黎、寓所、搬到、爱丁堡、布鲁塞尔\\ \hline 伦敦 + 美国 - 英国& 纽约、洛杉矶、伦敦、芝加哥、旧金山、亚特兰大\\ \hline 广州 + 浙江 - 广东& 杭州、宁波、嘉兴、金华、湖州、上海\\ \hline 广州 + 江苏 - 广东& 常州、无锡、苏州、南京、镇江、扬州\\ \hline 中学 + 大学生 - 大学& 中学生、中小学生、青少年、电子设计、村官、二中\\ \hline 人民币 + 美国 - 中国& 美元、港币、约合、美金、贬值、万美元\\ \hline 兵马俑 + 敦煌 - 西安& 莫高窟、卷子、写本、藏经洞、精美绝伦、千佛洞\\ \hline \end{array}\]
There's another point worth making here: some word analogy experiments look beautiful, and some look unconvincing, but in fact, word vectors reflect the statistical regularities of the corpus—they are objective. What's not objective, on the contrary, are some of the relations humans define. For a word embedding model, two words being close in vector space means they have similar context distributions, not that we've artificially defined them as similar. So whether the result looks good or not depends on how much the notion "similar context distribution ⇆ similar words" (which depends on the corpus) differs from the human definition of similarity (which is corpus-independent and subjective). When the experimental results don't look good, it's worth thinking along these lines.
Ranking Related Words
Recall equation $(15)$: the mutual information between two words equals the inner product of their word vectors. The larger the mutual information, the more likely the two words are to co-occur; the smaller it is, the less likely they are to ever be used together. So we can use inner-product ranking to find related words for a given word. Of course, the inner product also factors in the norm, and as we just said, the norm represents the importance of a word. If we disregard importance and consider only the meaning of the word itself, we should normalize the vectors first before taking the inner product—this approach is more stable:
\[\cos\theta_{ij}=\left\langle \frac{\boldsymbol{v}_i}{|\boldsymbol{v}_i|}, \frac{\boldsymbol{v}_j}{|\boldsymbol{v}_j|}\right\rangle\tag{30}\]
From probability theory, we know that if the mutual information is 0, the joint probability of two words is exactly equal to the probability of their random co-occurrence, meaning the two words are unrelated. Translated into equation $(15)$, this means the inner product of the two word vectors is 0, and from linear algebra we know that two vectors with zero inner product are mutually orthogonal—and we usually say that two orthogonal vectors are "unrelated." So, quite elegantly, statistical independence between two words corresponds exactly to geometric orthogonality. This is one of the beautiful things about the form of this model.
It should be pointed out that, as mentioned earlier, stopwords tend to shrink their norm rather than adjust their direction, so their direction carries essentially no meaning—we can consider a stopword's direction to be effectively random. This means that when we use cosine similarity to look for related words, we might occasionally get some surprising stopwords showing up.
Redefining Similarity
Note that what we discussed above was ranking related words—related words are not the same thing as synonymous words! For instance, "single" (单身) and "turned into" (冻成) are both strongly related to "dog" (狗), but they are not synonyms of "dog"; "science" (科学) is strongly related to "development concept" (发展观), but they aren't synonyms either.
So how do we find synonyms? In fact, this question puts the cart before the horse, because the definition of "similar" is itself a human construct. For example, "like" (喜欢) and "adore" (喜爱) are similar—but what about "like" (喜欢) and "dislike" (讨厌)? In a general topic-classification task they might be considered similar, but in a sentiment-classification task they are opposites. Or take "run" (跑) and "grab" (抓): normally we'd say these aren't similar, but in a part-of-speech classification task they are similar, since they share the same part of speech.
Let's go back to the assumption underlying our word embedding model: that a word's context distribution reveals its meaning. So, two similar words should have similar context distributions. The "airport − airplane + train = train station" example discussed earlier is based on the same principle, but there we required the context words to correspond exactly one-to-one, whereas here we only need approximate correspondence—a looser condition. And to accommodate different levels of similarity, we can even choose the context ourselves. Concretely, for two given words $w_i,w_j$ with corresponding word vectors $\boldsymbol{v}_i,\boldsymbol{v}_j$, to compute their similarity we first write out their mutual information with a pre-specified set of $N$ words, i.e.
\[\langle\boldsymbol{v}_i,\boldsymbol{v}_1\rangle,\langle\boldsymbol{v}_i,\boldsymbol{v}_2\rangle,\dots,\langle\boldsymbol{v}_i,\boldsymbol{v}_N\rangle\tag{31}\]
and
\[\langle\boldsymbol{v}_j,\boldsymbol{v}_1\rangle,\langle\boldsymbol{v}_j,\boldsymbol{v}_2\rangle,\dots,\langle\boldsymbol{v}_j,\boldsymbol{v}_N\rangle\tag{32}\]
Here $N$ is the total number of words in the vocabulary. If these two words are similar, their context distributions should also be similar, so the two sequences above should exhibit linear correlation. So it makes sense to compare them via the Pearson product-moment correlation coefficient:
\[\frac{\sum_{k=1}^N \Big(\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}\Big)\Big(\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle}\Big)}{\sqrt{\sum_{k=1}^N \Big(\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}\Big)^2}\sqrt{\sum_{k=1}^N \Big(\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle}\Big)^2}}\tag{33}\]
where $\overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}$ is the mean of $\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle$, i.e.
\[\overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}=\frac{1}{N}\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle=\left\langle\boldsymbol{v}_i,\frac{1}{N}\sum_{k=1}^N \boldsymbol{v}_k\right\rangle = \langle\boldsymbol{v}_i,\bar{\boldsymbol{v}}\rangle\tag{34}\]
so the correlation coefficient formula simplifies to
\[\frac{\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle\langle\boldsymbol{v}_j,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle}{\sqrt{\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle^2}\sqrt{\sum_{k=1}^N \langle\boldsymbol{v}_j,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle^2}}\tag{35}\]
In matrix notation (assuming all vectors here are row vectors), we have
\[\begin{aligned}&\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle\langle\boldsymbol{v}_j,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle\\ =&\sum_{k=1}^N \boldsymbol{v}_i (\boldsymbol{v}_k-\bar{\boldsymbol{v}})^{\top}(\boldsymbol{v}_k-\bar{\boldsymbol{v}})\boldsymbol{v}_j^{\top}\\ =&\boldsymbol{v}_i \left[\sum_{k=1}^N (\boldsymbol{v}_k-\bar{\boldsymbol{v}})^{\top}(\boldsymbol{v}_k-\bar{\boldsymbol{v}})\right]\boldsymbol{v}_j^{\top}\end{aligned}\tag{36}\]
What operation is happening inside that bracket? In fact it's just
\[\boldsymbol{V}^{\top}\boldsymbol{V},\,\boldsymbol{V}=\begin{pmatrix}\boldsymbol{v}_1-\bar{\boldsymbol{v}}\\ \boldsymbol{v}_2-\bar{\boldsymbol{v}}\\ \vdots \\ \boldsymbol{v}_N-\bar{\boldsymbol{v}}\end{pmatrix}\tag{37}\]
That is, we arrange the mean-subtracted word vectors into a matrix $\boldsymbol{V}$, and then compute $\boldsymbol{V}^{\top}\boldsymbol{V}$, which is a $n\times n$ real symmetric matrix, with $n$ being the dimensionality of the word vectors. It can be decomposed (Cholesky decomposition) as
\[\boldsymbol{V}^{\top}\boldsymbol{V}=\boldsymbol{U}\boldsymbol{U}^{\top}\tag{38}\]
where $\boldsymbol{U}$ is a real $n\times n$ matrix, so the correlation coefficient formula can be written as
\[\frac{\boldsymbol{v}_i \boldsymbol{U}\boldsymbol{U}^{\top}\boldsymbol{v}_j^{\top}}{\sqrt{\boldsymbol{v}_i \boldsymbol{U}\boldsymbol{U}^{\top}\boldsymbol{v}_i^{\top}}\sqrt{\boldsymbol{v}_j \boldsymbol{U}\boldsymbol{U}^{\top}\boldsymbol{v}_j^{\top}}}=\frac{\langle\boldsymbol{v}_i \boldsymbol{U},\boldsymbol{v}_j \boldsymbol{U}\rangle}{\Vert\boldsymbol{v}_i \boldsymbol{U}\Vert \times \Vert\boldsymbol{v}_j \boldsymbol{U}\Vert}\tag{39}\]
We find that similarity is still measured by the cosine of the vectors—only now we first transform them by the matrix $\boldsymbol{U}$ before taking the cosine.
Finally, how should we choose these $N$ words? We can sort words by descending frequency and take the top $N$. If $N$ is chosen fairly large (say, $N=10000$), we get related words in the general semantic sense—roughly the same as the results from the previous section. If $N$ is chosen small, such as $N=500$, we get syntactically similar words—for instance, in this case "爬" (crawl) turns out to be close to "掏" (dig out), "捡" (pick up), and "摸" (touch/feel), all sharing the same part of speech.
Keyword Extraction
As in The Unbelievable Word2Vec, Part 3: Extracting Keywords, a keyword is a word that can summarize the meaning of a sentence—meaning that just by looking at the keywords, one could roughly guess the overall content of the sentence. Suppose a sentence consists of $k$ words $w_1,w_2,\dots,w_k$; then the keywords should be those that maximize
\[P(w_1,w_2,\dots,w_k|w)\sim \frac{P(w_1,w_2,\dots,w_k;w)}{P(w_1,w_2,\dots,w_k)P(w)}\tag{40}\]
Put simply, this is the probability of guessing the sentence from the word being maximal. And since the sentence is given in advance, $P(w_1,w_2,\dots,w_k)$ is a constant, so maximizing the left-hand side is equivalent to maximizing the right-hand side. Continuing to use the naive assumption, from equation $(6)$ we get
\[\frac{P(w_1,w_2,\dots,w_k;w)}{P(w_1,w_2,\dots,w_k)P(w)}=\frac{P(w_1,w)}{P(w_1)P(w)}\frac{P(w_2,w)}{P(w_2)P(w)}\dots \frac{P(w_k,w)}{P(w_k)P(w)}\tag{41}\]
Substituting our word embedding model gives
\[e^{\langle\boldsymbol{v}_1,\boldsymbol{v}_w\rangle}e^{\langle\boldsymbol{v}_2,\boldsymbol{v}_w\rangle}\dots e^{\langle\boldsymbol{v}_k,\boldsymbol{v}_w\rangle}=e^{\left\langle\sum_i \boldsymbol{v}_i, \boldsymbol{v}_w\right\rangle}\tag{42}\]
so in the end this is equivalent to maximizing
\[\left\langle\sum_i \boldsymbol{v}_i, \boldsymbol{v}_w\right\rangle\tag{43}\]
The problem is now simple: given an input sentence, sum the word vectors of all its words to get a sentence vector, then take the inner product of the sentence vector with each individual word vector in the sentence (one could also use cosine similarity to get a normalized result), and sort in descending order. Simple and crude, and it reduces what would originally be an $\mathcal{O}(k^2)$-complexity algorithm down to $\mathcal{O}(k)$. How does it perform? Here are some examples.
Sentence: The mobilization meeting for the Second Central Environmental Protection Inspection Group's inspection work in Zhejiang Province was held in Hangzhou. From August 11 to September 11, the Central Environmental Protection Inspection Group formally moved in to carry out its work in Zhejiang. This also signals that, over the coming month, all enterprises in Zhejiang will face environmental inspections from the Central Environmental Protection Inspection Group, meaning they may face the risk of production cuts, suspensions, or shutdowns.
Keyword ranking: inspection group, production cuts, inspection group, mobilization meeting, shutdown, inspection, suspension, signal, environmental protection, imminent
Sentence: The Environmental Protection Bureau of Yiwu City, Zhejiang Province, stated that because the cadmium content in alloy raw materials is generally high, in order to control cadmium pollution, it has ordered some electroplating enterprises in the city to suspend production for rectification, and notified enterprises must suspend production for rectification effective immediately. It is reported that Yiwu's low-temperature zinc alloy (zinc-cadmium alloy) production has essentially halted. In addition, unapproved electroplating enterprises in Ouhai District, Wenzhou, have also been notified to unconditionally stop production starting August 18, and may only resume production after passing inspection. A new round of environmental protection efforts is underway among zinc downstream enterprises in Zhejiang.
Keyword ranking: zinc alloy, environmental protection bureau, suspend production, Ouhai District, approve, cadmium, electroplating, order, Yiwu City, raw materials
Sentence: Beef bourguignon is a classic French delicacy, described as "the most delicious beef that humankind can cook." This dish has a rich wine aroma and an inviting color, and the preparation process is not too complicated. What story lies behind it? How do you make delicious beef bourguignon?
Keyword ranking: beef bourguignon, wine aroma, delicacy, delicious, inviting, Burgundy, rich, dish, cooking, one (dish)
Sentence: Astronomy experts say that this year's meteor shower will reach its peak around 00:30 on the 18th, with a zenith hourly rate of about 10 meteors, including some bright fireballs, and it can be observed with the naked eye across most regions of China and indeed most of the Northern Hemisphere. This year's best observation window is from the early hours of the 17th to the 19th, and luckily there will be no moonlight interference at that time, which is favorable for observation.
Keyword ranking: meteor shower, fireball, Northern Hemisphere, zenith, observe, midnight, naked eye, at that time, count (of meteors), early morning
We can see that even for long sentences, this approach holds up quite well. It's worth noting that, although this method is simple and crude, this keyword-extraction approach doesn't work for every kind of word embedding—GloVe embeddings, for instance, don't work this way, because their stopwords have larger norms, so for GloVe the result is exactly reversed: the smaller the inner product (or cosine), the more likely a word is to be a keyword.
Sentence Similarity
Let's look at one more example—the question of sentence similarity, which many readers care about. In fact it's quite similar to keyword extraction.
When are two sentences similar, or even semantically equivalent? Simply put, it's when reading the first sentence tells you what the second sentence is saying, and vice versa. In such cases, the two sentences will necessarily have a high degree of correlation. Let sentence $S_1$ consist of $k$ words $w_1,w_2,\dots,w_k$, and sentence $S_2$ consist of $l$ words $w_{k+1},w_{k+2},\dots,w_{k+l}$. Using the naive assumption and equation $(6)$, we get
\[\frac{P(S_1,S_2)}{P(S_1)P(S_2)}=\prod_{i=1}^k\prod_{j=k+1}^{k+l} \frac{P(w_i,w_j)}{P(w_i)P(w_j)}\tag{44}\]
Substituting our word embedding model gives
\[\begin{aligned}\frac{P(S_1,S_2)}{P(S_1)P(S_2)}=&\prod_{i=1}^k\prod_{j=k+1}^{k+l} \frac{P(w_i,w_j)}{P(w_i)P(w_j)}\\ =&e^{\sum_{i=1}^k\sum_{j=k+1}^{k+l}\langle\boldsymbol{v}_i,\boldsymbol{v}_j\rangle}\\ =&e^{\left\langle\sum_{i=1}^k\boldsymbol{v}_i,\sum_{j=k+1}^{k+l}\boldsymbol{v}_j\right\rangle} \end{aligned}\tag{45}\]
so ultimately this is equivalent to ranking by
\[\left\langle\sum_{i=1}^k\boldsymbol{v}_i,\sum_{j=k+1}^{k+l}\boldsymbol{v}_j\right\rangle\tag{46}\]
The final result is likewise simple: just add up all the word vectors of each sentence to get its respective sentence vector, then take the inner product of the two sentence vectors (again, one could use cosine similarity for a normalized result), and this gives the correlation between the two sentences.
Sentence Vectors
The previous two sections both hinted that summing word vectors directly gives a sentence vector—so how good is the quality of such a sentence vector?
We ran a simple experiment: using sentence vectors obtained by summing word vectors (the non-truncated version) plus a linear classifier (logistic regression), we achieved around 81% accuracy on a sentiment classification task. If we add a hidden layer in between—input 128 (the word vector dimension; since the sentence vector is a sum of word vectors, it naturally has the same dimension), hidden layer 64 (ReLU activation), output 1 (binary classification)—we get around 88% accuracy. By comparison, LSTM achieves around 90% accuracy, so this kind of sentence vector performs quite creditably. Keep in mind that the word vectors used in this experiment were trained on the Baidu Baike corpus, meaning there was no sentiment information baked in from the start—yet it still managed to successfully and concisely capture the sentiment orientation of words.
At the same time, to verify the effect of weight truncation on the quality of this vector, we repeated the above experiment using the truncated version of the word vectors. The result was a maximum accuracy of 82% for logistic regression, and for the same three-layer neural network, a maximum accuracy of 89%. This shows that truncation (i.e., significantly down-weighting high-frequency words) does indeed help capture semantics better.
import pandas as pd
import jieba
pos = pd.read_excel('pos.xls', header=None)
neg = pd.read_excel('neg.xls', header=None)
pos[1] = pos[0].apply(lambda s: jieba.lcut(s, HMM=False))
neg[1] = neg[0].apply(lambda s: jieba.lcut(s, HMM=False))
pos[2] = pos[1].apply(w2v.sent2vec) #这个w2v.sentvec函数请参考下一篇
neg[2] = neg[1].apply(w2v.sent2vec)
pos = np.hstack([np.array(list(pos[2])), np.array([[1] for i in pos[2]])])
neg = np.hstack([np.array(list(neg[2])), np.array([[0] for i in neg[2]])])
data = np.vstack([pos, neg])
np.random.shuffle(data)
from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(64, input_shape=(w2v.word_size,), activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
batch_size = 128
model.fit(data[:16000,:w2v.word_size], data[:16000,[w2v.word_size]],
batch_size=batch_size,
epochs=100,
validation_data=(data[16000:,:w2v.word_size], data[16000:,[w2v.word_size]]))
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.