【The Amazing Word2Vec】 1. Mathematical Principles

【不可思议的Word2Vec】 1.数学原理 苏剑林 · 2017-04-02 · Machine Learning word vector word2vec language model

For readers familiar with deep learning and natural language processing (NLP), Word2Vec is a household name. Even if not everyone has actually used it, everyone should have at least heard of it — Google's efficient tool for obtaining word embeddings.

Is Word2Vec really amazing?

Most people treat Word2Vec as synonymous with "word embeddings," that is, purely as a tool for obtaining word vectors, and few readers pay much attention to the model itself. Perhaps because the model is so simplified, people assume that such a simplified model surely can't be very accurate, and hence isn't directly usable — yet its byproduct, the word embeddings, turn out to be surprisingly good. Indeed, judged purely as a language model, Word2Vec is far too crude.

But why should we look at it as a language model in the first place? If we set aside the constraints of language-model thinking and just look at the model itself, we find that Word2Vec's two models — CBOW and Skip-Gram — actually have far broader uses. They describe, from different angles, the relationship between surrounding words and the current word, and many fundamental NLP tasks are built on exactly this relationship, such as keyword extraction and logical reasoning. This series of articles is meant to serve as a starting point: by introducing the Word2Vec model itself, along with several seemingly "unbelievable" applications, I hope to offer some new ideas for studying this kind of problem. more

Speaking of Word2Vec's "amazing" side, when Word2Vec was first released, perhaps the most surprising thing was its word-analogy property — the linear relationship exemplified by king − man ≈ queen − woman. Mikolov, the author, believed that this property showed that the word vectors produced by Word2Vec possessed semantic reasoning ability, and it was precisely this property, combined with Google's halo effect, that made Word2Vec take off so quickly. Unfortunately, when we train our own word vectors, it's actually quite hard to reproduce this result, and there is no solid evidence that a good set of word embeddings should necessarily satisfy this word-analogy property. By contrast, the several applications I'll introduce here are highly reproducible — readers can train a Word2Vec model even on a small corpus and still obtain similar results.

Mathematical principles: online resources

Readers who want to understand this series properly should first get familiar with the mathematical principles behind Word2Vec. Of course, Word2Vec has been around for several years now, and there are countless articles explaining it. Here I recommend the blog series by the great peghoty:

http://blog.csdn.net/itplus/article/details/37969519

Also, this blog's own post What Exactly Are Word Vectors and Embeddings? can help us understand the principles of Word2Vec.

For readers' convenience, I've also collected two corresponding PDF files:

word2vector中的数学原理详解.pdf

Deep Learning 实战之 word2vec.pdf

The first one is the PDF version of peghoty's recommended blog series. Of course, if your English is good, you can go straight to Word2Vec's original papers:

[1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.
[2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.

Personally, though, I feel the original papers aren't explained as clearly as the Chinese sources.

Mathematical principles: a simple explanation

Put simply, Word2Vec is "two training schemes plus two speedup tricks," so strictly speaking it comprises four candidate models.

The two training schemes are CBOW and Skip-Gram, as shown in the figure

Word2Vec's two models

In plain language, these are "summing up the surrounding words to predict the current word" ($P(w_t|Context)$) and "using the current word to predict each surrounding word" ($P(w_{others}|w_t)$) — in other words, this is a conditional probability modeling problem. The two speedup tricks are hierarchical softmax and negative sampling. Hierarchical softmax simplifies softmax, reducing the cost of computing the predicted probability directly from $\mathcal{O}(|V|)$ down to $\mathcal{O}(\log_2 |V|)$, though it's somewhat less accurate than the original softmax. Negative sampling takes the opposite approach: it combines the original input and output together as a single input, and then performs binary classification to score it. This can be viewed as modeling the joint probabilities $P(w_t,Context)$ and $P(w_t,w_{others})$ — positive samples are those that actually occur in the corpus, while negative samples are randomly drawn. For more details, it's best to carefully read peghoty's blog series, which is also where I learned the implementation details of Word2Vec myself.

Finally, I should point out that the model used throughout this series is the combination of "Skip-Gram + hierarchical softmax" — that is, we will be using the $P(w_{others}|w_t)$ model itself, not merely the resulting word embeddings. So readers who want to follow the rest of this series need some familiarity with the Skip-Gram model, and should have some impression of how hierarchical softmax is constructed and implemented.

Stay tuned~