What Word Vectors and Embeddings Are Really About
Word vectors, known in English as Word Embedding, literally translate to "word embedding" — that much is clear. When people hear "word vector," many readers will immediately think of Google's Word2Vec; that's the brand-name effect at work. Additionally, frameworks like Keras have an Embedding layer, which is also described as mapping word IDs to vectors. Because of this preconception, people tend to equate word vectors with Word2Vec, and then ask the reverse question — "which kind of word vector is Embedding?" — which, especially for beginners, is bound to be confusing. In fact, even veterans don't always manage to explain it clearly.
It all starts with one-hot encoding...
The Pot Calling the Kettle Black
One-hot encoding is the most primitive way of representing characters or words. For simplicity, this post uses individual characters as examples; words work the same way. Suppose our vocabulary contains six characters: 科, 学, 空, 间, 不, 错. One-hot encoding assigns each of these six characters a 0-1 encoding:
$$\begin{array}{c|c}\hline\text{subject} & [1, 0, 0, 0, 0, 0]\\ \text{learn} & [0, 1, 0, 0, 0, 0]\\ \text{empty} & [0, 0, 1, 0, 0, 0]\\ \text{between} & [0, 0, 0, 1, 0, 0]\\ \text{not} & [0, 0, 0, 0, 1, 0]\\ \text{wrong} & [0, 0, 0, 0, 0, 1]\\ \hline \end{array}$$more
So, to represent the word "科学" (science), we could use the matrix
$$\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0 & 0 \end{pmatrix}$$
You've probably already spotted the problem: however many characters there are, that's how many dimensions the vector needs — if there are 10,000 characters, each character vector would be 10,000-dimensional (commonly used characters might not number that many, maybe a few thousand, but if we're talking about words rather than characters, common words could easily number in the hundreds of thousands). Hence the introduction of continuous vector representations — say, using a 100-dimensional real-valued vector to represent a character, which greatly reduces the dimensionality and the risk of overfitting, and so on. That's what beginners say, and quite a few experts say it too.
But the truth is: nonsense! Nonsense! Nonsense! Important enough to say three times.
Let me give you a little test to make this clear: give someone two arbitrary 100×100 real-valued matrices and ask them to compute the product — probably very few people could do it by hand. But give someone two 1000×1000 matrices, where one of them is one-hot (each row has exactly one 1 and the rest 0s), and ask them to multiply them, and they'll figure it out quickly. Don't believe me? Try it yourself.
See the issue now? One-hot matrices may be huge, but they're easy to compute with. Your fancy low-dimensional real-valued matrix, despite being smaller, is actually more of a hassle to multiply (although this amount of computation is nothing for a computer). Of course, there's a deeper reason too, which comes below.
Seemingly Wrong, Actually Right
Let's actually compute
$$\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0 & 0 \end{pmatrix}\begin{pmatrix}w_{11} & w_{12} & w_{13}\\ w_{21} & w_{22} & w_{23}\\ w_{31} & w_{32} & w_{33}\\ w_{41} & w_{42} & w_{43}\\ w_{51} & w_{52} & w_{53}\\ w_{61} & w_{62} & w_{63}\end{pmatrix}=\begin{pmatrix}w_{11} & w_{12} & w_{13}\\ w_{21} & w_{22} & w_{23}\end{pmatrix}$$
The left-hand side shows this is a fully-connected neural network layer taking a 2×6 one-hot matrix as input, with 3 hidden units. But look at the right-hand side — isn't that exactly the same as taking rows 1 and 2 from the matrix $w_{ij}$? And isn't that exactly the same as the so-called "table lookup" for character vectors (finding the vector corresponding to a given character in a table)? Indeed it is! This is precisely the so-called Embedding layer. An Embedding layer is simply a fully-connected layer that takes one-hot vectors as input and has as many hidden units as the dimensionality of the character vectors! And the parameters of this fully-connected layer are exactly the "character vector table"! From this perspective, character vectors don't do anything special at all! They are one-hot — stop mocking one-hot for its supposed problems, because character vectors are literally the parameters of the fully-connected layer applied to one-hot vectors!
So does that mean character vectors and word vectors bring no innovation whatsoever? They do bring something. From a computational standpoint, people essentially discovered that multiplying by a one-hot matrix is equivalent to a table lookup, so they directly implement it as a lookup operation instead of writing it out as a matrix multiplication, which greatly reduces the computational cost. Let me stress again: the reduction in computation doesn't come from the introduction of word vectors per se, but from simplifying one-hot matrix multiplication into a table-lookup operation. That's the computational side. On the conceptual side, once you have the parameters of this fully-connected layer, you directly use these parameters as features — or in other words, you use the parameters of this fully-connected layer as the representation of characters or words, thereby obtaining character/word vectors. And it turns out these vectors have some interesting properties, such as cosine similarity between vectors capturing, to some extent, the semantic similarity between characters or words.
By the way, some people criticize Word2Vec (CBOW) for being just a three-layer model, not deserving the label "deep" learning. But actually, if you count the fully-connected layer applied to the one-hot input, that makes four layers — which is arguably a modest little deep model after all.
Where Do They Come From?
Wait — if word vectors are treated as the parameters of a fully-connected layer (dear reader, let me correct that: they're not merely "treated as" such — they literally are such), then you still haven't told me how these parameters are obtained! The answer is: I don't know either. Aren't the parameters of a neural network determined by your task? You should be asking yourself about the task, not me! You might say Word2Vec is unsupervised? Let me clarify that too.
Strictly speaking, neural networks are all supervised in some sense, and models like Word2Vec are, more precisely, "self-supervised." What it actually trains is a language model, and word vectors are obtained via this language model. A language model, roughly speaking, predicts the probability of the next character given the preceding $n$ characters — it's just a multi-class classifier. We feed in one-hot vectors, connect them to a fully-connected layer, then stack a few more layers, and finally attach a softmax classifier — that gives us a language model. Feed it a large amount of text for training, and in the end the parameters of that first fully-connected layer become the character/word vector table. Of course, Word2Vec makes a lot of simplifications, but those simplifications are all made to the language model itself; its first layer is still a fully-connected layer, and the parameters of that fully-connected layer are still the character/word vector table.
Seen this way, the picture becomes much simpler — there's no particular reason you have to train vectors via a language model, right? Indeed, you can use other tasks, for example supervised training via a text sentiment classification task. Because, as already stated, it's just a fully-connected layer — what you attach after it is entirely up to you. Of course, since labeled data is generally scarce, this approach is prone to overfitting, so people usually first pretrain character/word vectors on large-scale unlabeled corpora to reduce the risk of overfitting. Note carefully: the reason overfitting risk is reduced is that unlabeled corpora can be used for pretraining word vectors (unlabeled corpora can be huge, and with enough data there's no overfitting risk) — it has nothing to do with word vectors themselves. A word vector is just a layer of parameters to be trained; how could it possibly have any inherent power to reduce overfitting risk?
Finally, let's explain why these character/word vectors end up having properties such that cosine similarity or Euclidean distance between vectors can, to some degree, reflect semantic similarity between characters or words. This is because, during unsupervised training of the language model, we operate within a window: predicting the probability of the next character from the preceding $n$ characters, where $n$ is the window size. Words appearing within the same window receive similar updates, and these updates accumulate over training; words with similar usage patterns will accumulate these similar updates to a noticeable degree. Let me give an example: the characters "忐" and "忑" are almost always used together; updating "忐" almost always comes with updating "忑" as well, so their updates are nearly identical, and as a result the vectors for "忐" and "忑" inevitably end up nearly the same. "Similar usage patterns" here means that, within a specific language task, the words are interchangeable — for example, in general-purpose corpora, replacing "喜欢" (like) with "讨厌" (dislike) in a sentence like "我喜欢你" (I like you), or in most other contexts, still yields a valid sentence. Hence "喜欢" and "讨厌" inevitably end up with similar word vectors. But if the word vectors are instead trained via a sentiment classification task, then "喜欢" and "讨厌" would end up with quite different word vectors.
To Be Continued, Sort Of
It feels like there's still more to say, yet also like there's nothing more worth adding. Hopefully this piece helps clarify the concepts of character and word vectors, and helps you get to the bottom of what one-hot and Embedding really are. If you have questions or new insights, please feel free to leave a comment.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.