【The Amazing Word2Vec】 2. A Pretrained Model

【不可思议的Word2Vec】 2.训练好的模型 苏剑林 · 2017-04-03 · Machine Learning word vector word2vec language model

Since the next few posts will discuss how to use Word2Vec, I've gone ahead and trained a Word2Vec model in advance. To save readers time, and to make sure everyone can reproduce the results in later posts, I've decided to share this pretrained model, which was trained with Gensim. The word vectors alone aren't very large, but as mentioned in the first post, we need the complete Word2Vec model, so I'm sharing the full model, which consists of four files — hence the relatively larger size.

A reminder for readers: if you want to obtain a complete Word2Vec model without modifying the source code, Python's Gensim library is really your only option. As far as I know, other implementations of Word2Vec only ever give you the word vectors, not the complete model.

For knowledge mining purposes, a Word2Vec model trained on knowledge-base corpora (such as encyclopedia text) would obviously perform better. I'm still in the process of crawling encyclopedia corpora, though — once that's done, I'll train another model and share it then.

Overview of the model

Here's a rough overview of this model:

$$\begin{array}{c|c} \hline \text{training corpus} & \text{WeChat articles, multi-domain, Chinese balanced corpus}\\ \hline \text{corpus size} & \text{8M docs, 65B words total}\\ \hline \text{model vocab size} & \text{共352196词，基本是中文词，包含常见英文词}\\ \hline \text{model structure} & \text{Skip-Gram + Huffman Softmax}\\ \hline \text{vector dimension} & \text{256-dim}\\ \hline \text{tokenizer} & \text{结巴分词，加入了有50万词条的词典，关闭了新词发现}\\ \hline \text{training tool} & \text{Gensim Word2Vec, trained 7 days on server}\\ \hline \text{other cases} & \text{window size 10, min freq 64, 10 iterations}\\ \hline \end{array}$$more

One thing worth pointing out specifically: WeChat public account articles are fairly "contemporary" in nature — they reflect recent internet hot topics and have fairly broad coverage, so they're a reasonably representative sample of text. For tokenization, I used Jieba, with new-word discovery turned off — the philosophy being that it's better to under-segment than to segment inaccurately. Of course, the built-in dictionary alone isn't sufficient, so I compiled an additional 500,000 entries myself, from two sources: 1) merging dictionaries collected from the web; 2) running new-word discovery on the public account articles and manually vetting the results before adding them to the dictionary. As a result, the tokenization output is reasonably reliable and includes quite a lot of popular/trending words, making it fairly usable.

Training code

Feel free to adapt this for your own use. Note that the hashlib.md5 import here is used for deduplicating articles (starting from 10 million articles, deduplication brought it down to 8 million) — this step isn't strictly necessary.

#! -*- coding:utf-8 -*-

import gensim, logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

import pymongo
import hashlib

db = pymongo.MongoClient('172.16.0.101').weixin.text_articles_words
md5 = lambda s: hashlib.md5(s).hexdigest()
class sentences:
    def __iter__(self):
        texts_set = set()
        for a in db.find(no_cursor_timeout=True):
            if md5(a['text'].encode('utf-8')) in texts_set:
                continue
            else:
                texts_set.add(md5(a['text'].encode('utf-8')))
                yield a['words']
        print u'最终计算了%s篇文章'%len(texts_set)

word2vec = gensim.models.word2vec.Word2Vec(sentences(), size=256, window=10, min_count=64, sg=1, hs=1, iter=10, workers=25)
word2vec.save('word2vec_wx')

Download link

Link: https://pan.baidu.com/s/1htC495U Password: 4ff8
Includes the files: word2vec_wx, word2vec_wx.syn1neg.npy, word2vec_wx.syn1.npy, word2vec_wx.wv.syn0.npy — all four files are required for Gensim to load the model. I haven't fully worked out what each file represents: word2vec_wx is presumably the model declaration, word2vec_wx.wv.syn0.npy should be what we call the word vector table, word2vec_wx.syn1.npy is the hidden-to-output layer parameters (the Huffman tree parameters), and word2vec_wx.syn1neg.npy I'm not too sure about...
If you only care about the word vectors, you can also download the C-format version (compatible with the C implementation of Word2Vec, containing only the word vectors):
Link: https://pan.baidu.com/s/1nv3ANLB Password: dgfw

A few demos

Just casually demonstrating this model's results for finding synonyms/related words below. Suggestions for improvement are welcome.

import gensim
model = gensim.models.Word2Vec.load('word2vec_wx')
pd.Series(model.most_similar(u'微信'))
0 (QQ, 0.752506196499)
1 (订阅号, 0.714340209961)
2 (QQ号, 0.695577561855)
3 (扫一扫, 0.695488214493)
4 (微信公众号, 0.694692015648)
5 (私聊, 0.681655049324)
6 (微信公众平台, 0.674170553684)
7 (私信, 0.65382117033)
8 (微信平台, 0.65175652504)
9 (官方, 0.643620729446)
pd.Series(model.most_similar(u'公众号'))
0 (订阅号, 0.782696723938)
1 (微信公众号, 0.760639667511)
2 (微信公众账号, 0.73489522934)
3 (公众平台, 0.716173946857)
4 (扫一扫, 0.697836577892)
5 (微信公众平台, 0.696847081184)
6 (置顶, 0.666775584221)
7 (公共账号, 0.665741920471)
8 (微信平台, 0.661035299301)
9 (菜单栏, 0.65234708786)
pd.Series(model.most_similar(u'牛逼'))
0 (牛掰, 0.701575636864)
1 (厉害, 0.619165301323)
2 (靠谱, 0.588266670704)
3 (苦逼, 0.586573541164)
4 (吹牛逼, 0.569260418415)
5 (了不起, 0.565731525421)
6 (牛叉, 0.563843131065)
7 (绝逼, 0.549570798874)
8 (说真的, 0.549259066582)
9 (两把刷子, 0.545115828514)
pd.Series(model.most_similar(u'广州'))
0 (东莞, 0.840889930725)
1 (深圳, 0.799216389656)
2 (佛山, 0.786817133427)
3 (惠州, 0.779960036278)
4 (珠海, 0.73523247242)
5 (厦门, 0.72509008646)
6 (武汉, 0.724122405052)
7 (汕头, 0.719602584839)
8 (增城, 0.713532209396)
9 (上海, 0.710560560226)
pd.Series(model.most_similar(u'朱元璋'))
0 (朱棣, 0.857951819897)
1 (燕王, 0.853199958801)
2 (朝廷, 0.847517609596)
3 (明太祖朱元璋, 0.837111353874)
4 (赵匡胤, 0.835654854774)
5 (称帝, 0.835589051247)
6 (起兵, 0.833530187607)
7 (明太祖, 0.829249799252)
8 (太祖, 0.826784193516)
9 (丞相, 0.826457977295)
pd.Series(model.most_similar(u'微积分'))
0 (线性代数, 0.808522999287)
1 (数学分析, 0.791161835194)
2 (高等数学, 0.786414265633)
3 (数学, 0.758676528931)
4 (概率论, 0.747221827507)
5 (高等代数, 0.737897276878)
6 (解析几何, 0.730488717556)
7 (复变函数, 0.715447306633)
8 (微分方程, 0.71503329277)
9 (微积分学, 0.704192101955)
pd.Series(model.most_similar(u'apple'))
0 (banana, 0.79927945137)
1 (pineapple, 0.789698243141)
2 (pen, 0.779583632946)
3 (orange, 0.769554674625)
4 (sweet, 0.721074819565)
5 (fruit, 0.71402490139)
6 (pie, 0.711439430714)
7 (watermelon, 0.700904607773)
8 (apples, 0.697601020336)
9 (juice, 0.694036960602)
pd.Series(model.most_similar(u'企鹅'))
0 (海豹, 0.665253281593)
1 (帝企鹅, 0.645192623138)
2 (北极熊, 0.619929730892)
3 (大象, 0.618502140045)
4 (鲸鱼, 0.606555819511)
5 (猫, 0.591019570827)
6 (蜥蜴, 0.584576964378)
7 (蓝鲸, 0.572826981544)
8 (海豚, 0.566122889519)
9 (猩猩, 0.563284397125)
pd.Series(model.most_similar(u'足球'))
0 (篮球, 0.842746257782)
1 (足球运动, 0.819511592388)
2 (青训, 0.793446540833)
3 (排球, 0.774085760117)
4 (乒乓球, 0.760577201843)
5 (足球赛事, 0.758624792099)
6 (棒垒球, 0.750351667404)
7 (篮球运动, 0.746055066586)
8 (足球队, 0.74296438694)
9 (网球, 0.742858171463)
pd.Series(model.most_similar(u'爸爸'))
0 (妈妈, 0.779690504074)
1 (儿子, 0.752222895622)
2 (奶奶, 0.70418381691)
3 (妈, 0.693783283234)
4 (爷爷, 0.683066487312)
5 (父亲, 0.673043072224)
6 (女儿, 0.670304119587)
7 (爸妈, 0.669358253479)
8 (爸, 0.663688421249)
9 (外婆, 0.652905225754)
pd.Series(model.most_similar(u'淘宝'))
0 (淘, 0.770935535431)
1 (店铺, 0.739198565483)
2 (手机端, 0.728774428368)
3 (天猫店, 0.725838780403)
4 (口令, 0.721312999725)
5 (登录淘宝, 0.717839717865)
6 (淘宝店, 0.71473968029)
7 (淘宝搜, 0.697688698769)
8 (天猫, 0.690212249756)
9 (网店, 0.6820114851)