【The Incredible Word2Vec】4. Different Kinds of "Similarity"
Defining Similarity
Once we obtain word vectors via Word2Vec, we typically use cosine similarity to compare how similar two words are, defined as
$$\cos (\boldsymbol{x}, \boldsymbol{y}) = \frac{\boldsymbol{x}\cdot\boldsymbol{y}}{|\boldsymbol{x}|\times|\boldsymbol{y}|}$$
With this notion of similarity in hand, we can compare the similarity between any two given words, and also find the words most similar to a given word. In gensim's Word2Vec, this is implemented by the most_similar function.
Wait a moment! We've quickly written down the formula for computing similarity, but we haven't actually "defined" similarity at all! How can we arrive at a mathematical formula for evaluating similarity without even having a definition of similarity itself?
This is not a question we can casually skip over. Very often we go ahead and do things without really knowing what it is we're doing. Take the previous article on keyword extraction as an example — I suspect many people never stopped to ask what a keyword actually is. Is it merely a word that is somehow "key"? But once we do stop and think about it, we realize a keyword is something used to estimate roughly what an article is about, which gives us a natural definition of a keyword:
$$keywords = \mathop{\text{argmax}}_{w\in s}p(s|w)$$
and from there we can model it using various methods.
Coming back to the topic of this article: how should we define similarity? The answer is: it depends on the scenario — you define whatever notion of similarity the situation calls for. more
So then, what kind of similarity does cosine similarity actually give us? As it happens, Word2Vec, at its core, describes a word using the average distribution of its context (since Word2Vec ignores word order), and cosine value has nothing to do with vector norm — so what it captures is "relative consistency." A high cosine similarity essentially means that the two words frequently pair up with the same set of surrounding words, or, put more crudely, that within the same sentence the two words are mutually substitutable. For instance, the words most similar to "Guangzhou" are "Dongguan" and "Shenzhen" — this is because, in many contexts, directly swapping "Guangzhou" for "Dongguan" or "Shenzhen" in a sentence still yields a grammatically sound sentence (sound as a sentence, though not necessarily true as a fact — e.g. "Guangzhou is the capital of Guangdong" becomes "Dongguan is the capital of Guangdong," which is a well-formed sentence, but not a true statement).
\>>> s = u'广州'
\>>> pd.Series(model.most_similar(s))
0 (东莞 [Dongguan], 0.840889930725)
1 (深圳 [Shenzhen], 0.799216389656)
2 (佛山 [Foshan], 0.786817014217)
3 (惠州 [Huizhou], 0.779960155487)
4 (珠海 [Zhuhai], 0.735232532024)
5 (厦门 [Xiamen], 0.725090026855)
6 (武汉 [Wuhan], 0.724122405052)
7 (汕头 [Shantou], 0.719602525234)
8 (增城 [Zengcheng], 0.713532149792)
9 (上海 [Shanghai], 0.710560560226)
Relatedness: A Different Kind of Similarity
As already mentioned, the definition of similarity really depends on the scenario, and cosine similarity is only one option. Sometimes we feel that "Dongguan" and "Guangzhou" have essentially no real connection — for an old-time Guangzhou local, words like "Baiyun Mountain," "Baiyun Airport," and "Canton Tower" are the ones that feel truly "similar" to "Guangzhou." This kind of scenario is also quite common — for example, in tourism recommendation, after a tourist arrives in Guangzhou, we'd naturally want the system, upon input of "Guangzhou," to output related terms like "Baiyun Mountain," "Baiyun Airport," and "Canton Tower," rather than words like "Dongguan" or "Shenzhen."
This kind of "similarity" is, more precisely, "relatedness" — how should we characterize it? The answer is mutual information, defined as
$$\log \frac{p(x,y)}{p(x)p(y)}=\log p(y|x) - \log p(y)$$
The larger the mutual information, the more frequently the two words $x,y$ co-occur.
This way, given a word $x$, we can find the words that frequently co-occur with $x$, and this can also be done entirely with Word2Vec's Skip-Gram + Huffman Softmax model. The code is as follows:
import numpy as np
import gensim
model = gensim.models.word2vec.Word2Vec.load('word2vec_wx')
def predict_proba(oword, iword):
iword_vec = model[iword]
oword = model.wv.vocab[oword]
oword_l = model.syn1[oword.point].T
dot = np.dot(iword_vec, oword_l)
lprob = -sum(np.logaddexp(0, -dot) + oword.code*dot)
return lprob
from collections import Counter
def relative_words(word):
r = {i:predict_proba(i, word)-np.log(j.count) for i,j in model.wv.vocab.iteritems()}
return Counter(r).most_common()
The resulting related words for "Guangzhou" are:
\>>> s = u'广州'
\>>> w = relative_words(s)
\>>> pd.Series(w)
0 (福中路 [Fuzhong Road], -17.390365773)
1 (OHG, -17.4582544641)
2 (林寨镇 [Linzhai Town], -17.6119545612)
3 (坪山街道 [Pingshan Subdistrict], -17.6462214199)
4 (东圃镇 [Dongpu Town], -17.6648893759)
5 (西翼 [West Wing], -17.6796614955)
6 (北京西 [Beijing West], -17.6898282385)
7 (⇋, -17.6950761384)
8 (K1019, -17.7259853233)
9 (景泰街道 [Jingtai Subdistrict], -17.7292421556)
10 (PSW3, -17.7296432222)
11 (广州铁路职业技术学院 [Guangzhou Railway Polytechnic], -17.732288911)
12 (13A06, -17.7382891287)
13 (5872, -17.7404719442)
14 (13816217517, -17.7650583156)
15 (未遂案 [attempted case], -17.7713452536)
16 (增城市 [Zengcheng City], -17.7713832873)
17 (第十甫路 [Dishifu Road], -17.7727940473)
18 (广州白云机场 [Guangzhou Baiyun Airport], -17.7897457043)
19 (Faust, -17.7956389314)
20 (国家档案馆 [National Archives], -17.7971039916)
21 (w0766fc, -17.8051687721)
22 (K1020, -17.8106548248)
23 (陈宝琛 [Chen Baochen], -17.8427718407)
24 (jinriGD, -17.8647825023)
25 (3602114109100031646, -17.8729896156)
As you can see, the results obtained are, on the whole, closely related to Guangzhou. Of course, sometimes we may want to give slightly more weight to high-frequency words, so we can revise the mutual information formula to:
$$\log \frac{p(x,y)}{p(x)p^{\alpha}(y)}=\log p(y|x) - \alpha\log p(y)$$
where $\alpha$ is a constant slightly less than 1. If we take $\alpha=0.9$, then we get:
from collections import Counter
def relative_words(word):
r = {i:predict_proba(i, word)-0.9*np.log(j.count) for i,j in model.wv.vocab.iteritems()}
return Counter(r).most_common()
Re-sorting the results gives:
\>>> s = u'广州'
\>>> w = relative_words(s)
\>>> pd.Series(w)
0 (福中路 [Fuzhong Road], -16.8342976099)
1 (北京西 [Beijing West], -16.9316053191)
2 (OHG, -16.9532688634)
3 (西翼 [West Wing], -17.0521852934)
4 (增城市 [Zengcheng City], -17.0523156839)
5 (广州白云机场 [Guangzhou Baiyun Airport], -17.0557270208)
6 (林寨镇 [Linzhai Town], -17.0867272184)
7 (⇋, -17.1061883426)
8 (坪山街道 [Pingshan Subdistrict], -17.1485480457)
9 (5872, -17.1627067119)
10 (东圃镇 [Dongpu Town], -17.192150594)
11 (PSW3, -17.2013228493)
12 (Faust, -17.2178736991)
13 (红粉 [pink/rouge], -17.2191157626)
14 (国家档案馆 [National Archives], -17.2218467278)
15 (未遂案 [attempted case], -17.2220391092)
16 (景泰街道 [Jingtai Subdistrict], -17.2336594498)
17 (光孝寺 [Guangxiao Temple], -17.2781121397)
18 (国际货运代理 [international freight forwarding], -17.2810157155)
19 (第十甫路 [Dishifu Road], -17.2837591345)
20 (广州铁路职业技术学院 [Guangzhou Railway Polytechnic], -17.2953441257)
21 (芳村 [Fangcun], -17.301106775)
22 (检测院 [inspection institute], -17.3041253252)
23 (K1019, -17.3085465963)
24 (陈宝琛 [Chen Baochen], -17.3134413583)
25 (林和西 [Linhe West], -17.3150577006)
Relatively speaking, this second set of results is somewhat more readable. Here are some other examples:
\>>> s = u'飞机' [airplane]
\>>> w = relative_words(s)
\>>> pd.Series(w)
0 (澳门国际机场 [Macau International Airport], -16.5502216186)
1 (HawkT1, -16.6055740672)
2 (架飞机 [-planes, measure word], -16.6105400944)
3 (地勤人员 [ground crew], -16.6764712234)
4 (美陆军 [US Army], -16.6781627384)
5 (SU200, -16.6842796275)
6 (起降 [takeoff and landing], -16.6910345896)
7 (上海浦东国际机场 [Shanghai Pudong International Airport], -16.7040362134)
8 (备降 [diversion landing], -16.7232609719)
9 (第一架 [the first (aircraft)], -16.7304077856)
\>>> pd.Series(model.most_similar(s))
0 (起飞 [take off], 0.771412968636)
1 (客机 [passenger jet], 0.758365988731)
2 (直升机 [helicopter], 0.755871891975)
3 (一架 [one (aircraft)], 0.749522089958)
4 (起降 [takeoff and landing], 0.726713418961)
5 (降落 [land], 0.723304390907)
6 (架飞机 [-planes, measure word], 0.722024559975)
7 (飞行 [flight], 0.700125515461)
8 (波音 [Boeing], 0.697083711624)
9 (喷气式飞机 [jet aircraft], 0.696866035461)
\>>> s = u'自行车' [bicycle]
\>>> w = relative_words(s)
\>>> pd.Series(w)
0 (骑 [ride], -16.4410312554)
1 (放风筝 [fly a kite], -16.6607225423)
2 (助力车 [power-assisted bike], -16.8390451582)
3 (自行车 [bicycle], -16.900188791)
4 (三轮车 [tricycle], -17.1053629907)
5 (租赁点 [rental point], -17.1599389605)
6 (电动车 [electric bike], -17.2038996636)
7 (助动车 [moped], -17.2523149342)
8 (多辆 [multiple vehicles], -17.2629832083)
9 (CRV, -17.2856425014)
\>>> pd.Series(model.most_similar(s))
0 (摩托车 [motorcycle], 0.737690329552)
1 (骑 [ride], 0.721182465553)
2 (滑板车 [scooter], 0.7102201581)
3 (电动车 [electric bike], 0.700758457184)
4 (山地车 [mountain bike], 0.687280654907)
5 (骑行 [cycling], 0.666575074196)
6 (单车 [bicycle], 0.651858925819)
7 (骑单车 [riding a bicycle], 0.650207400322)
8 (助力车 [power-assisted bike], 0.635745406151)
9 (三轮车 [tricycle], 0.630989730358)
Feel free to try this yourself. One caveat: unfortunately, while Huffman Softmax speeds up computation during training, at prediction time — when we need to traverse the entire vocabulary — it is actually slower than plain Softmax, so this is not an efficient approach in practice.
What Have We Actually Done Here?
From the two parts above, we can see that "similarity" generally comes in two flavors: (1) frequently pairing up with the same set of surrounding words, and (2) frequently co-occurring together. Both can be regarded as forms of similarity between words, each suited to different needs.
For example, when disambiguating the sense of a polysemous word — say, whether "star" means "celestial body" or "celebrity" — mutual information can be put to use. We can first gather a corpus where "star" is used in the sense of "celestial body," and find the words with high mutual information with "star" in that corpus; these might include words like sun, planet, earth. Likewise, we can gather a corpus where "star" means "celebrity," and find the words with high mutual information with "star" there; these might include words like entertainment, movie. Then, given a new context, we can use these co-occurrence patterns to infer which sense is intended.
In short, the key is to first pin down exactly what you need, and only then choose the appropriate method.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.