Is a higher singular-value entropy of matrix parameters always better?

矩阵参数的奇异值熵越高越好吗？ 苏剑林 · 2026-05-29 · Mathematics matrix entropy SVD mean field

In last year's technical report Muon is Scalable for LLM Training, in order to compare the differences between models trained with Muon and Adam, we introduced the concept of "singular-value entropy," and observed that the matrix parameters trained by Muon generally had higher singular-value entropy than those trained by Adam. We took this as empirical evidence that "Muon makes fuller use of the model parameters."

But is a higher singular-value entropy really always better? Is there an optimal singular-value entropy? Or, to turn the question around: if we had to fix the singular-value entropy of a matrix parameter in advance, what value should we choose?

Recap of Concepts

Let's first recall a few concepts. We know that the singular values of a matrix are all non-negative, so we can normalize them in some way—for instance via direct normalization or squared normalization:

\begin{equation}p_i = \frac{\sigma_i}{\sum_{j=1}^n \sigma_j}\qquad\text{or}\qquad p_i = \frac{\sigma_i^2}{\sum_{j=1}^n \sigma_j^2}\end{equation}more

This gives us an $n$-way probability distribution $\boldsymbol{p} = (p_1,p_2,\cdots,p_n)$. Once we have a probability distribution, we can compute related quantities, such as the entropy

\begin{equation}H(\boldsymbol{p}) = -\sum_{i=1}^n p_i \log p_i\end{equation}

This is the notion of "singular-value entropy." We know that maximum entropy corresponds to a uniform distribution, so singular-value entropy also measures how evenly distributed the singular values are. However, entropy is not a single unique concept—the log-expectation form above is what we call "Shannon entropy," but there are other kinds of entropy, such as the Rényi entropy

\begin{equation}H_q(\boldsymbol{p}) = -\frac{1}{q-1}\log \sum_{i=1}^n p_i^q \end{equation}

In fact, Shannon entropy is a special case of it, corresponding to $q\to 1$. For other values of $q$, it likewise attains its maximum $\log n$ at the uniform distribution and its minimum $0$ at the one-hot distribution. The singular-value entropy of a matrix is essentially equivalent to the effective rank of the matrix; another way to interpret it is as a measure of how sparse the matrix's singular values are (see How to Measure the Sparsity of Data?).

Reformulating the Problem

The question now is: how do we turn "is a higher singular-value entropy really always better" into a computable proposition, so as to give a reference answer?

Intuitively, since current models are generally over-parameterized, there is a good deal of redundant freedom—which allows us to impose some constraints on the model parameters (such as constraining their norm or their singular-value entropy) without noticeably hurting performance. Once a constraint is imposed on the parameters, the remaining degrees of freedom are naturally reduced. We can empirically assume: the larger the remaining degrees of freedom, the stronger the model's expressive capacity—this is the principle behind our search for an "optimal singular-value entropy."

So, given a fixed singular-value entropy, how many degrees of freedom does the matrix retain? Let's first look at two extreme cases. The first is minimum entropy, corresponding to only one nonzero singular value; in this case the matrix is a rank-1 matrix, with degrees of freedom no more than $2n$. The second is maximum entropy, corresponding to all nonzero singular values being equal; here the matrix is (a scalar multiple of) an orthogonal matrix, with degrees of freedom of about $n^2/2$. From this it seems maximum entropy is indeed better than minimum entropy—but is there an even better value in between?

We can also set aside the matrix context and look directly at the "entropy density" of singular-value distributions. For an $n$-way distribution, the maximum entropy is $\log n$, attained only by the uniform distribution; the minimum entropy is $0$, corresponding to a one-hot distribution, of which there are $n$ different choices. Clearly, the "density" at both boundaries looks rather low—they can only represent a small fraction of all distributions—so we might guess that the optimum indeed lies somewhere in between.

Reformulated this way, our problem becomes: if we uniformly sample an $n$-way distribution and compute its entropy, what does the distribution of the entropy look like? Where is the probability density maximized? The point of maximum probability density means that in its neighborhood there is the largest number of distributions, and we take this as the entropy value with the greatest expressive capacity.

A Geometric Picture

Some readers might be puzzled by the idea of "uniformly sampling an $n$-way distribution"—normally we're given a probability distribution and then sample a number from it, but here we want to sample the probability distribution itself. However, if we temporarily set aside the probabilistic meaning of $\boldsymbol{p}$, we can get a very clear geometric picture.

Specifically, an $n$-way probability distribution $\boldsymbol{p}$ is really a $n$-dimensional vector satisfying the following constraints:

\begin{equation}0 \leq p_i \leq 1 \, (\forall i=1,2,\cdots,n)\qquad \sum_{i=1}^n p_i = 1\end{equation}

The first constraint describes a unit hypercube in a $n$-dimensional space, while the second describes a hyperplane in a $n$-dimensional space. Their intersection gives a bounded piece of that hyperplane, which is called the "$n-1$-simplex," denoted $\Delta^{n-1}$. So "uniformly sampling an $n$-way distribution" is really just uniformly sampling a point on this piece of hyperplane—which makes the whole idea much more intuitive.

Readers might wonder why the triangle symbol $\Delta$ is used to denote a simplex. In fact, when $n=3$, this piece of hyperplane is exactly the equilateral triangle with vertices $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$; and it's not hard to imagine that when $n=4$, it forms a regular tetrahedron in three-dimensional space. Continuing this pattern, we see that geometrically, a simplex is really the higher-dimensional generalization of an equilateral triangle—hence the notation $\Delta$.

A Limit Theorem

Back to the main thread. Let's now consider computing the probability density of the entropy, which can be formally written as

\begin{equation}\rho(h) = \frac{\int_{\boldsymbol{p}\in\Delta^{n-1}} \delta(H(\boldsymbol{p}) - h) d\boldsymbol{p}}{\int_{\boldsymbol{p}\in\Delta^{n-1}} d\boldsymbol{p}}\end{equation}

where $\delta(\cdot)$ is the Dirac delta function. The physical meaning of this expression is quite intuitive: it "counts" the distributions with a given $H(\boldsymbol{p})=h$ and then divides by the total count. However, this is purely a formal solution—computing it analytically in practice is nearly impossible.

Given that in real scenarios $n$ is usually fairly large (hundreds or even thousands), we can invoke the "central limit theorem" and assume that $\rho(h)$ is approximately normally distributed. A normal distribution has only two parameters—mean and variance—so we only need to compute the mean and variance (the second moment) of the entropy, i.e., $\mathbb{E}_{\boldsymbol{p}\sim\Delta^{n-1}}[H(\boldsymbol{p})]$ and $\mathbb{E}_{\boldsymbol{p}\sim\Delta^{n-1}}[H(\boldsymbol{p})^2]$. If we're only after the point of maximum probability, we only need the mean, since for a normal distribution the point of maximum probability coincides with the mean.

The difficulty in computing the mean $\mathbb{E}_{\boldsymbol{p}\sim\Delta^{n-1}}[H(\boldsymbol{p})]$ lies in how to realize the sampling of $\boldsymbol{p}\sim\Delta^{n-1}$—while this sampling is conceptually intuitive, it's not obvious how to implement it computationally. If readers are familiar with some background on the Dirichlet distribution, they can use some existing results to speed up understanding, but here we deliberately avoid introducing the Dirichlet distribution, to keep the barrier to entry lower for general readers.

Reformulating the Sampling

Fortunately, uniform sampling on a simplex can be reformulated as independently and repeatedly sampling from an exponential distribution $\text{Exp}(1)$:

\begin{equation}\boldsymbol{p}\sim\Delta^{n-1}\qquad\Leftrightarrow\qquad p_i = \frac{x_i}{\sum_{j=1}^n x_j},\quad x_1,x_2,\cdots,x_n\sim \text{Exp}(1)\end{equation}

That is, independently and repeatedly sampling $n$ numbers from the exponential distribution $\text{Exp}(1)$ and then normalizing them is equivalent to sampling uniformly on the simplex. How should we understand this? Of course it can be proven rigorously, but here I'd rather introduce an "isotropy analogy" perspective to make sense of it.

Let's switch to a different question first: uniformly sampling a $n$-dimensional unit vector. This isn't hard to do—just sample $n$ numbers $(x_1,x_2,\cdots,x_n)=\boldsymbol{x}$ from a standard normal distribution $\mathcal{N}(0,1)$, and then return $\boldsymbol{x}/\Vert\boldsymbol{x}\Vert$. Why does this work? Because the probability density of $\mathcal{N}(0,1)$ is proportional to $e^{-x^2/2}$, and sampling $n$ times gives $e^{-\sum_{i=1}^n x_i^2/2} = e^{-\Vert\boldsymbol{x}\Vert^2/2}$, which depends only on the norm of $\boldsymbol{x}$ and not on its direction—so sampling from $\mathcal{N}(0,1)$ is uniform with respect to direction.

A unit vector is a vector whose "sum of squares equals 1." The probability distribution we want to sample now is instead a vector whose "sum equals 1"—so we can think of a probability distribution as a unit vector with respect to the "sum" as its norm. And we know that the probability density of the exponential distribution $\text{Exp}(1)$ is $e^{-x}$, and sampling $n$ times gives $e^{-\sum_{i=1}^n x_i}$, which depends only on the "sum" norm—so it is uniform with respect to this new kind of "direction vector."

Mean-Field Analysis

Having set all this up, we can now get down to the actual computation. Under this new parametrization, the entropy $H(\boldsymbol{p})$ can be transformed into

\begin{equation}H(\boldsymbol{p}) = \log \sum_{i=1}^n x_i - \frac{\sum_{i=1}^n x_i \log x_i}{\sum_{i=1}^n x_i}\end{equation}

Taking the expectation of both sides then gives us a rather complicated integral. Interestingly, although this integral is complicated, it can in fact be solved analytically using certain tricks. We won't dig into those details here, though—instead, we'll use a mean-field trick to quickly get an approximate solution:

\begin{equation}\begin{aligned} \mathbb{E}\left[\log \sum_{i=1}^n x_i - \frac{\sum_{i=1}^n x_i \log x_i}{\sum_{i=1}^n x_i}\right] \approx \log \sum_{i=1}^n \mathbb{E}[x_i] - \frac{\sum_{i=1}^n \mathbb{E}[x_i \log x_i]}{\sum_{i=1}^n \mathbb{E}[x_i]} = \log n - (1 - \gamma) \end{aligned}\end{equation}

where $\gamma$ is the Euler–Mascheroni constant. The result $\mathbb{E}[x_i]=1$ is obvious, and the slightly harder part is $\mathbb{E}[x_i \log x_i]=1-\gamma$, but this is also a classical result related to the derivative of the Gamma function; even without knowing this result, one can simply compute it with Mathematica.

We know that the maximum entropy of an $n$-way distribution is $\log n$, and now we have an extra term $-(1-\gamma) = - 0.42278\cdots$, which is exactly what we were after: it shows that the point of maximum entropy density lies roughly at a value smaller than the maximum entropy by $0.42$. For a matrix, its expressive capacity is stronger for singular-value entropies near this point, rather than simply as high as possible. If we need to fix the singular-value entropy to some value, we can choose this one.

The General Result

Using the same approach, we can compute the expectation of the general Rényi entropy:

\begin{equation}\begin{aligned} \mathbb{E}[H_q(\boldsymbol{p})] =&\, \frac{1}{q-1}\mathbb{E}\left[q\log \sum_{i=1}^n x_i -\log \sum_{i=1}^n x_i^q\right] \\ \approx&\, \frac{1}{q-1}\left[q\log \sum_{i=1}^n \mathbb{E}[x_i] - \log \sum_{i=1}^n \mathbb{E}[x_i^q]\right] \\ =&\, \frac{1}{q-1}\left[q \log n - \log n\Gamma(q+1)\right] \\ =&\, \log n - \frac{\log \Gamma(q+1)}{q-1} \end{aligned}\label{eq:E-Hq}\end{equation}

If $q\to 1$, the corresponding limit is exactly $\log n - (1-\gamma)$, consistent with the result of the previous section. If $q$ is an integer greater than 1, we can directly write

\begin{equation}\mathbb{E}[H_q(\boldsymbol{x})] \approx \log n - \frac{\log q\,!}{q-1}\end{equation}

If we take $q=2$, the result simplifies elegantly to $\log (n/2)$. We know that $\log (n/2)$ is the maximum entropy of an $n/2$-way distribution, so the intuitive interpretation of this result is: the singular-value entropy only needs to "fill up" half the dimensions—there's no need to fill up all of them, since doing so would actually sacrifice diversity.

A Complete Analysis

In the previous two sections, we focused on estimating the mean, which is already sufficient for our original purpose. However, for the sake of completeness, in this section we'll add an estimate of the variance, and also discuss why asymptotic normality holds for $\rho(h)$.

First, let's transform $H_q(\boldsymbol{p})$ into an identity:

\begin{equation}H_q(\boldsymbol{p}) = \log n + \frac{1}{q-1}\Bigg[q\log \underbrace{\frac{1}{n}\sum_{i=1}^n x_i}_u -\log \underbrace{\frac{1}{n}\sum_{i=1}^n x_i^q}_v\Bigg]\end{equation}

When $q\neq 1$, $u,v$ are two distinct statistics; the central limit theorem tells us that as $n\to\infty$, $(u,v)$ asymptotically follows a bivariate normal distribution. From $x_i\sim \text{Exp}(1)$, we can derive that the mean and variance of $u$ are $1$ and $1/n$ respectively, and the mean and variance of $v$ are $\Gamma(q+1)$ and $[\Gamma(2q+1) - \Gamma(q+1)^2]/n$ respectively.

This shows that when $q$ is not too large and $n$ is large, both quantities concentrate tightly around their respective means, which allows us to do a Taylor expansion around the mean—for example, a first-order approximation

\begin{equation}H_q(\boldsymbol{p}) \approx \log n + \frac{1}{q-1}\Bigg[q(u-1) - \left(\log \Gamma(q+1) + \frac{v - \Gamma(q+1)}{\Gamma(q+1)}\right)\Bigg]\end{equation}

Taking the expectation of both sides recovers formula $\eqref{eq:E-Hq}$; if needed, we could also expand to higher order to get the remainder term. Since $H_q(\boldsymbol{p})$ is now linear in $u,v$, we can compute the variance using the following formula

\begin{equation}\mathbb{V}\mathrm{ar}[H_q(\boldsymbol{p})] \approx \mathbb{V}\mathrm{ar}\left[\frac{q u}{q-1}\right] + \mathbb{V}\mathrm{ar}\left[\frac{v}{(q-1) \Gamma(q+1)}\right] - 2\mathbb{C}\mathrm{ov}\left[\frac{q u}{q-1},\frac{v}{(q-1) \Gamma(q+1)}\right]\end{equation}

Note that here $u,v$ are generally not independent, so the covariance term cannot be neglected

\begin{equation}\mathbb{C}\mathrm{ov}[u, v] = \frac{1}{n^2}\sum_{i=1}^n \sum_{j=1}^n \mathbb{C}\mathrm{ov}[x_i, x_j^q] = \frac{1}{n^2}\sum_{i=1}^n \mathbb{C}\mathrm{ov}[x_i, x_i^q]\end{equation}

The first equality follows from the definition of $u,v$ and the bilinearity of $\mathbb{C}\mathrm{ov}$; the second equality holds because when $i\neq j$, $x_i,x_j$ are mutually independent, so the covariance is 0. Computing further gives $\mathbb{C}\mathrm{ov}[x_i, x_i^q] = \mathbb{E}[x_i^{q+1}] - \mathbb{E}[x_i] \mathbb{E}[x_i^q]=\Gamma(q+2) - \Gamma(q+1) = q \Gamma(q+1)$. Substituting these results, we get

\begin{equation}\mathbb{V}\mathrm{ar}[H_q(\boldsymbol{p})] \approx \frac{1}{n}\left[\frac{\Gamma(2q+1)}{(q-1)^2 \Gamma(q+1)^2} - \frac{q^2+1}{(q-1)^2}\right]\end{equation}

When $q=2$, the result is $1/n$; when $q\to 1$, the result is $(\pi^2/3-3)/n$.

In short, the central limit theorem does not directly give us asymptotic normality of the entropy $h$—we can only obtain asymptotic normality of the intermediate variable $u,v$, and from there estimate the mean and variance of $h$. This shows that the variance of $h$ is also $\mathcal{O}(1/n)$, meaning the variance shrinks as $n$ grows—so it is indeed sound to locate the point of maximum probability density via the mean.

Summary

This article turns the question of "is a higher singular-value entropy always better" into a quantitatively analyzable mathematical proposition. The core assumption is: once the singular-value entropy is fixed, the remaining degrees of freedom of a matrix parameter represent its expressive capacity. We then interpreted the point of maximum degrees of freedom as the point of maximum entropy density, and through a series of transformations and approximations, arrived at an optimal entropy value.

Of course, the result itself isn't especially important—the real value of this article lies more in providing a reference framework for analyzing related questions.