How was DeepSeek V4's tid2eid derived?

DeepSeek V4的tid2eid是怎么来的？ 苏剑林 · 2026-05-15 · Machine Learning model function analysis MoE

Anyone who has trained an MoE model knows that if you replace the entire MLP part of the model with MoE, the first few MoE layers near the embedding tend to be very difficult to load-balance. To deal with this, DeepSeek V3, including our own Kimi K2, adopted the strategy of "first_k_dense" — as the name suggests, the first $k$ layers don't use MoE, but instead use a regular dense-type GLU. In DeepSeek V4, this strategy was changed to "first_k_hash".

Here "hash" refers to "Hash Routing", proposed in Hash Layers For Large Sparse Models, which assigns each token to an expert via a predetermined mapping table tid2eid. In this post I want to explore how this tid2eid table might be generated.more

Background

First, it should be pointed out that for the first few MoE layers, using the Quantile Balancing method the author proposed in MoE Journey: 6. Optimal Assignment for Balance and MoE Journey: 7. A Minimalist Solution for Dynamic Activation can already alleviate the imbalance problem to a considerable degree.

As for V4's adoption of Hash Routing, it can be seen as a different-direction attempt at the same problem. The idea is that for the first few MoE layers, there isn't much contextual information yet, so choosing an expert based on the input vector at this stage is perhaps not that different from choosing directly based on the token id, which carries no context at all. Given that, why not simply hard-code, ahead of time, which expert each token should activate in the first few layers, based purely on the token id? This is where "tid2eid" (Token Id to Expert Id) comes from.

So how should this tid2eid table be generated? Would a completely random assignment work? Doesn't seem like it, because different tokens occur with different frequencies — a fully random assignment would actually lead to imbalance. DeepSeek hasn't published the details of how this table is generated, so we can only take a guess based on our own reasoning.

Mathematical formulation

Suppose there are $m$ distinct tokens in total, and the frequency of the $i$-th token is $p_i$. Without loss of generality, assume they are already sorted in descending order, i.e. $p_i\geq p_{i+1}$. Let $x_{i,j}\in\{0, 1\}$ denote whether token $i$ should activate expert $j$ ($0$ means not activated, $1$ means activated), with a total of $n$ experts, and each token selecting $k$ experts. Then what we actually want to solve is the following system of equations:

\begin{equation}x_{i,j}\in\{0, 1\},\qquad\sum_{j=1}^n x_{i,j} = k,\qquad \sum_{i=1}^m p_i x_{i,j}\approx \frac{k}{n}\end{equation}

Note that in the last equality we used an approximate equality $\approx$, because if we instead wrote $=$, the system would strictly speaking not necessarily have a solution. So we keep the notation $\approx$, meaning that the two sides should be as close as possible. This can also be cast as an optimization problem:

\begin{equation}\min_{x_{i,j}\in\{0, 1\}} \sum_{j=1}^n \left(\sum_{i=1}^m p_i x_{i,j} - \frac{k}{n}\right)^2 \qquad\text{s.t.}\qquad\sum_{j=1}^n x_{i,j} = k\end{equation}

A fairly straightforward way to solve this is a greedy algorithm:

Process tokens one by one in order of decreasing frequency. For each token, first tally the current load already assigned to each expert, then pick the $k$ experts with the lightest load as the activated experts for the current token, and finally update the load distribution of the experts accordingly.

Reference implementation

Although the greedy algorithm is, in principle, greedy, in practice it can usually find a solution that is close to optimal. A reference implementation is as follows:

import numpy as np

# 模拟分布
m, n, k = 80000, 128, 4
p = 1 / (10 + np.arange(m))
p /= p.sum()

# 贪心处理
x, f = np.zeros((m, k), dtype='int32'), np.zeros(n)
for i in range(m):
    j = f.argsort()[:k]  # 选择负载最轻的k个
    f[j] += p[i] / k  # 更新负载分布
    x[i] = j  # 记录到tid2eid中

# 评估均衡
max_vio, min_vio = f.max() * n - 1, f.min() * n - 1

Note that this code contains no randomness at all — in principle it is a deterministic algorithm. So what if we want different tid2eid mappings for the first few layers? We could introduce some randomness — for instance, replacing range(m) with np.random.permutation(m), i.e. not processing tokens strictly in order of decreasing frequency. This still yields a usable solution, just with somewhat worse balance. We can run it multiple times and then pick the few solutions with the lowest max_vio.

An extreme case

Now let's consider an extreme scenario: $p_1 \gg k / n$, i.e. some token's frequency already far exceeds the balanced level. In this case, no matter how tid2eid is arranged, load balancing simply cannot be achieved. In other words, if the decision is based solely on a single token id, there is no way to achieve balance at all. How should we handle this situation?

The answer is simple: switch to a hashing method that depends on more input information. Previously we made decisions based only on the current token id — but in reality, every token sits within some sequence, so it has context. If the previous approach can be called "1-gram to expert", we could now consider combining it with the preceding token(s) to form "2-gram to expert", "3-gram to expert", and so on.

Take the 2-gram $(a, b)$ as an example. A straightforward hashing approach is: choose a prime $q$ larger than $m$, compute $(aq+b)\mod n$ as the activated expert, and repeat this $k$ times with different primes to obtain $k$ experts. Because increasing the number of grams greatly increases the number of input combinations, once these are mapped into a finite set of $n$ values, each value is very likely to get "filled up" evenly. So as long as the hash function isn't particularly bad, the result will be almost perfectly balanced.

The advantage of this, then, is that there's no need to worry about frequencies, nor to precompute and store a tid2eid table. The drawback is that computing the hash function is somewhat more complex, and the same token might end up selecting duplicate experts — but neither of these is a particularly serious issue.

Summary

This post briefly reviewed the basic idea behind Hash Routing in DeepSeek V4, with a particular focus on how its tid2eid mapping table might be constructed.