A More Elegant Word Vector Model (I): Simpler GloVe

更别致的词向量模型(一)：simpler glove 苏剑林 · 2017-11-19 · Machine Learning word vector GloVe

If you asked me which word vector model is the most convenient and practical, I'd say word2vec. But if you asked me which is the most elegant, I honestly wouldn't know — I feel that every model has something lacking. Setting aside whether the experimental results are good or bad (that's ultimately just a matter of evaluation metrics), even purely in terms of theory, no model really deserves to be called elegant.

This post discusses several questions about word vectors that many people care about. Most of the conclusions are largely empirical findings, lacking a solid theoretical explanation, including:

How should one go about constructing a word vector model?
Why do we use cosine similarity to search for near-synonyms? What does the inner product of vectors actually mean?
Does the norm of a word vector have any special meaning?
Why do word vectors exhibit analogy properties (king − man + woman = queen)?
Once we have word vectors, how do we build sentence vectors? What's the justification for using the sum of word vectors as a simple sentence vector?

These discussions are both specific and, in some sense, general — some of the explanations here might carry over directly to interpreting the properties of word vectors from GloVe and the skip-gram model, and readers are welcome to try that themselves.

Around these questions, this post proposes a new GloVe-like word vector model, which I'll call simpler GloVe. Building on and modifying Stanford's GloVe source code, I provide an implementation here; the actual code is on Github. more

Why bother improving GloVe? To be fair, GloVe's underlying idea is genuinely inspiring. However, despite claims that it rivals or even surpasses word2vec, GloVe itself is actually a rather clumsy model (we'll explain why later), and that leaves plenty of room for improvement.

Overview of contents:

1 Modeling language
1.1 From conditional probability to mutual information
1.2 The additivity of mutual information
1.3 A digression: bonus section
2 Describing correlations
2.1 Geometric word vectors
2.2 airport − airplane + train = train station
2.3 The form of the model
2.4 Forgetting normalization
3 Solving the model
3.1 Loss function
3.2 Estimating mutual information
3.3 Weighting and downsampling
3.4 Adagrad
4 Interesting results
4.1 The meaning of vector norm
4.2 Word analogy experiments
4.3 Ranking related words
4.4 Redefining similarity
4.5 Keyword extraction
4.6 Sentence similarity
4.7 Sentence vectors