A More Elegant Word Embedding Model (VI): Code, Sharing, and Closing Remarks

更别致的词向量模型(六)：代码、分享与结语 苏剑林 · 2017-11-19 · Machine Learning word vector GloVe

List

A More Elegant Word Embedding Model (I): Simpler GloVe
A More Elegant Word Embedding Model (II): Modeling Language
A More Elegant Word Embedding Model (III): Describing the Correlation Model
A More Elegant Word Embedding Model (IV): Solving the Model
A More Elegant Word Embedding Model (V): Interesting Results
A More Elegant Word Embedding Model (VI): Code, Sharing, and Closing Remarks

Code

The implementation for this post can be found at: https://github.com/bojone/simpler_glovemore

The source code is modified from Stanford's original GloVe. My changes are only minor, since the main difficulty lies in computing the co-occurrence word frequency statistics — many thanks to the researchers at Stanford for providing such a classic and excellent implementation of that statistical procedure. To be honest, I'm not very familiar with C, so the modifications I made might not be particularly elegant; I'd welcome corrections from anyone more experienced.

In addition, to reproduce the "interesting results" from the previous section, I've also included simpler_glove.py in the GitHub repo. It wraps a class that can directly load the model files (in txt format) exported by the C version of simpler GloVe, along with some commonly used helper functions for convenience.

Sharing

Here is a set of Chinese word vectors trained with the model described in this post. The corpus consists of about 1 million Baidu Baike articles, with a vocabulary of roughly 300,000 words, and the embedding dimension is 128. One special treatment was applied during tokenization: all digits and English letters were split into individual digits and letters. If anyone would like to experiment with these vectors, feel free to download them:

Link: http://pan.baidu.com/s/1jIb3yr8
Password: 1ogw

Closing Remarks

This series has been a fairly thorough exploration of word embedding models, and to some extent the product of my own theoretical obsessiveness. Fortunately, I ended up with a model that looks reasonably elegant in theory, which has at least partially cured that obsession. As for its experimental performance and practical applications, that will have to be verified through further use down the road.

Most of the derivations in this series can be adapted, in much the same way, to explain the experimental results of word2vec's skip-gram model — readers are encouraged to try this themselves. In fact, the skip-gram model of word2vec does behave quite similarly to the model presented here, including in terms of the properties of the resulting word vectors.

All in all, combining theory with experiments is a wonderful thing to do — though also, admittedly, a rather exhausting one, since just working through all of the above took me several months of thinking.