[The Incredible Word2Vec] 3. Extracting Keywords

【不可思议的Word2Vec】 3.提取关键词 苏剑林 · 2017-04-07 · Machine Learning word vector word2vec language model

This article gives a new definition of what a keyword is, and provides an implementation based on Word2Vec. This definition of "keyword" is natural and reasonable, and Word2Vec is just one simplified way to implement it — one could use the same definition with other models.

When people talk about extracting keywords, TF-IDF and TextRank usually come to mind. But has anyone ever thought about using Word2Vec to extract keywords? Not only that, but using Word2Vec to extract keywords already incorporates a degree of semantic understanding, rather than being purely statistical — and it's unsupervised too!

What is a keyword?

Admittedly, TF-IDF and TextRank are two classic algorithms for extracting keywords, and both have a certain rationale behind them. But the problem is, if you'd never seen these two algorithms before, you'd probably find their results rather far-fetched, and it would be hard to construct them from scratch on your own. In other words, although these two algorithms look simple, they aren't easy to come up with. Just imagine: a student who hasn't studied information theory would probably struggle to understand why IDF takes a logarithm. Why not some other function? And how many readers would have ever thought, out of the blue, of borrowing PageRank's idea to judge the importance of a word?

Ultimately, the issue is this: extracting keywords and summarizing text both seem like natural tasks, but has anyone really thought about what the definition of a "keyword" actually is? I don't mean looking it up in a Chinese dictionary and getting a wall of text — I mean, what is its mathematical definition? What should the mathematically sound definition of a keyword be? Or, put differently, what is the purpose of extracting keywords in the first place? more

Clearly, whether we're talking about keywords or summaries, what we want is to grasp the gist of an article as quickly as possible. If an article's keyword is "deep learning," we know it's unlikely to be a rant about "how steel is tempered." In other words, we should be able to guess the gist of a text from its keywords. Mathematically, this is expressed as the conditional probability

$$p(s|w_i)$$

Here $s$ represents a piece of text, and $w_i$ is some word in that text. If $w_i$ is a keyword of the text, then it should maximize the probability above. In other words, we just need to compute this probability for every word in the sentence, sort them in descending order, and we've extracted the keywords. Put simply, a keyword is the word that best lets us guess the original text. How do we estimate this probability? A simple naive Bayes assumption will do: if $s$ is made up of $n$ words $w_1,w_2,\dots,w_n$, then

$$p(s|w_i)=p(w_1,w_2,\dots,w_n|w_i)=\prod_{k=1}^n p(w_k|w_i)$$

This way, we only need to estimate the word-to-word transition probability $p(w_k|w_i)$ to obtain the conditional probability $p(s|w_i)$, which completes the keyword extraction.

What does this have to do with Word2Vec? To estimate $p(w_k|w_i)$, we need a large amount of text for statistics, and fortunately this process is unsupervised, so gathering statistics is straightforward. But we have an even better tool at our disposal — what tool is best suited for modeling $p(w_k|w_i)$? Readers may already have guessed: it's obviously Word2Vec! Isn't Word2Vec's Skip-Gram model precisely designed to model this probability? Given how "fast, accurate, and effective" Word2Vec is, there's no reason not to use it! (Of course, as stated at the beginning of the article, we don't necessarily have to use the Bayes assumption, and we certainly don't have to use Word2Vec to compute it — but the definition of a keyword itself should be sound and reasonable.)

Computing probabilities with Word2Vec

At this point, readers should understand why I emphasized the Skip-Gram + Huffman Softmax combination so heavily in the previous two articles — because this combination is exactly what models $p(w_k|w_i)$. Of course, due to the nature of Huffman Softmax, computing $p(w_k|w_i)$ takes a bit of extra work. Reference code below:

import numpy as np
import gensim
model = gensim.models.word2vec.Word2Vec.load('word2vec_wx')

def predict_proba(oword, iword):
    iword_vec = model[iword]
    oword = model.wv.vocab[oword]
    oword_l = model.syn1[oword.point].T
    dot = np.dot(iword_vec, oword_l)
    lprob = -sum(np.logaddexp(0, -dot) + oword.code*dot) 
    return lprob

This is basically written by referring directly to the score_sg_pair function in gensim's Word2Vec implementation. The gist of the process is: take the Huffman code (path) for $w_k$, take the word vector for $w_i$, and then, following the path, compute the probability at each node along the path and multiply them together to get $p(w_k|w_i)$. Since this actually computes the log-probability, the multiplication becomes addition. So how is the final probability computed? In fact, according to the Word2Vec formulas, the log-probability at each node is:

$$\begin{aligned}&\log \left(\frac{1}{1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}}\right)^{1-d}\left(1-\frac{1}{1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}}\right)^{d}\\ =&-(1-d)\log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \boldsymbol{x}^{\top} \boldsymbol{\theta}\\ =&-\log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \boldsymbol{x}^{\top} \boldsymbol{\theta}\end{aligned}$$

Here $\boldsymbol{\theta}$ is the node vector, $\boldsymbol{x}$ is the input word vector, and $d$ is the code (either 0 or 1) at that node. However, the official score_sg_pair function isn't written exactly this way, and that's because

$$\begin{aligned}&-\log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \boldsymbol{x}^{\top} \boldsymbol{\theta}\\ =&-\log \bigg[e^{d \boldsymbol{x}^{\top}\theta}(1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}})\bigg]\\ =&-\log \bigg(e^{d \boldsymbol{x}^{\top}\theta}+e^{(d-1)\boldsymbol{x}^{\top} \boldsymbol{\theta}}\bigg)\\ =&-\log \bigg(1+e^{-(-1)^d \boldsymbol{x}^{\top}\theta}\bigg)\end{aligned}$$

Practice makes perfect

With the groundwork above laid, computing keywords is now simple:

from collections import Counter
def keywords(s):
    s = [w for w in s if w in model]
    ws = {w:sum([predict_proba(u, w) for u in s]) for w in s}
    return Counter(ws).most_common()

import pandas as pd #引入它主要是为了更好的显示效果
import jieba
s = u'太阳是一颗恒星'
pd.Series(keywords(jieba.cut(s)))

The output is:

0 (star, -27.9013707845)
1 (sun, -28.1072913493)
2 (a, -30.482187911)
3 (is, -36.3372344659)

Other examples:

s=u'The Changping District government website states that the Ming Tombs are the world's best-preserved and most extensively populated imperial burial complex; they were designated as a National Key Cultural Heritage Site by the State Council in the first batch of 1961, and were inscribed on the World Heritage List in 2003.'
pd.Series(keywords(jieba.cut(s)))
0 (cultural heritage protection, -261.691625676)
1 (list, -272.297758506)
2 (world heritage, -273.943120665)
3 (first batch, -280.781786703)
4 (designated as, -281.663865896)
5 (Ming Tombs, -286.298893108)
6 (burial complex, -287.463013816)
...
s=u'The emergence of Xiong'an New Area attracted many out-of-town speculators to buy property there. However, as the local government cracked down on illegal property speculation and the housing market froze, investment demand that couldn't find an outlet in Xiong'an New Area spilled over into the surrounding areas.'
pd.Series(keywords(jieba.cut(s)))
0 (property speculators, -326.997266407)
1 (housing market, -336.176584187)
2 (property speculation, -337.190896137)
3 (buy property, -344.613473556)
4 (purchase property, -346.396359454)
5 (crack down, -350.207272082)
6 (out-of-town, -355.860419218)
s=u'If you're designing costumes for a period drama, you must consider which dynasty the story is set in — the Han dynasty favored loose robes with wide sleeves, while the Qing dynasty had mandarin jackets and qipao. Yet on the Peking opera stage, almost any historical figure, based on their gender, age, status, and basic character traits, can find a suitable costume among the existing repertoire.'
pd.Series(keywords(jieba.cut(s)))
0 (dynasty, -485.150966757)
1 (figure, -493.759615898)
2 (period costume, -495.478962392)
3 (Han dynasty, -503.409908377)
4 (Qing dynasty, -503.45656029)
5 (qipao, -504.76313228)
6 (status, -507.624260109)

Feel free to try it out yourself. If you want to try this on your own corpus, just train a Word2Vec model (Skip-Gram + Huffman Softmax) on your corpus, and then call the code above.

You probably have some doubts

Following our original idea, $p(w_k|w_i)$ ought to be computed by counting statistics over the whole sentence, whereas Word2Vec only uses a window. Is that reasonable? In fact, even though Word2Vec only uses a window, it has already successfully established connections between similar words. That is to say, when we use Word2Vec for the process above, we are effectively aggregating "similar words" for the evaluation, whereas TF-IDF only aggregates "identical words" for the evaluation. This is why we say that extracting keywords with Word2Vec already incorporates a degree of semantic judgment. Moreover, by taking $p(w_k|w_i)$ into account, Word2Vec also captures relationships within the text — there's a bit of a TextRank flavor here, since it's a bigram model, whereas TF-IDF only considers the information content of individual words and is merely a unigram model.

Furthermore, since Word2Vec is trained via a neural network, it comes with built-in smoothing: even if two words never co-occur in the text, we can still get a reasonably sensible probability for them.

Of course, this comes at a cost: the efficiency of the TF-IDF algorithm is $\mathcal{O}(N)$, whereas extracting keywords with Word2Vec has an efficiency of $\mathcal{O}(N^2)$, where $N$ is the number of words in the sentence.