Deriving Scaling Laws for Models from a Quantization Hypothesis
Scaling Law refers to the asymptotic relationship between model capability and model scale. Concretely, we can simply understand model capability as the model's loss function, and model scale can refer to the parameter count, the amount of training data, the number of training steps, and so on. The study of scaling laws is thus the study of how the loss function roughly relates to variables such as parameter count, data volume, and training steps. Experimental results from works such as Scaling Laws for Neural Language Models and Training Compute-Optimal Large Language Models show that neural network scaling laws mostly take the form of a "power law."
Why should it be a power law? Can this be explained theoretically? The paper The Quantization Model of Neural Scaling offers a rather interesting derivation based on a "quantization" hypothesis. Let's take a look at it together.
Assumptions for the Derivation
First, let's assume that for a given task there exists a "perfect model," and the model we actually train is an approximation of this "perfect model." Furthermore, assume the "perfect model" is composed of "quanta," where each quantum represents a certain capability (note that here, "quantum" mainly refers to an abstract unit of capability, not a specific, nameable skill).
Completing a task typically requires many kinds of capabilities, so without loss of generality, we assume the "perfect model" contains infinitely many capability quanta, with different quanta responsible for handling samples of different difficulty. Generally speaking, easy samples make up the majority while hard samples are a minority, so these capability quanta can be ranked from highest to lowest frequency of occurrence, denoted $1,2,\cdots,k,\dots$, with corresponding occurrence frequencies $p_1,p_2,\cdots,p_k,\cdots$.
Finally, we assume the frequencies of these capability quanta follow "Zipf's law," i.e.,
\begin{equation}p_k = \frac{k^{-\gamma - 1}}{Z_{\gamma}}\end{equation}
where $\gamma > 0$, and $Z_{\gamma}$ is the normalizing factor $\sum\limits_{k=1}^{\infty}k^{-\gamma - 1}$.
Zipf's Law
Readers might wonder: why Zipf's law? Zipf's law is an empirical law published by Zipf in 1949. His original finding was that the frequency of a word's occurrence is roughly inversely proportional to its rank in a frequency table. Later, this was generalized to being inversely proportional to a "power of the rank," and it turns out that Zipf's law can be observed in many different settings.
Zipf himself, along with later researchers, attempted to derive Zipf's law from assumptions closer to first principles; related work can be found on Wikipedia, so we won't go into it here. For the author, the main reason for choosing Zipf's law is honestly — there really isn't any other option.
Recall that $p_k$ has already been sorted from high to low, so $p_k$ is a monotonically decreasing function. What non-negative, monotonically decreasing functions can we think of? Basically just exponential functions and power functions. Exponential functions decay very quickly, so there's no long-tail phenomenon, whereas power functions decay more slowly and hence exhibit a longer tail. Which one to choose depends on our prior belief about how important the tail is. Given our earlier assumption about capability quanta — that every capability matters — we can only choose the power function, and the result is Zipf's law.
The Basic Result
Back to the main topic. Earlier we assumed the ideal model has infinitely many capability quanta, but for a real-world model with finite capacity, it can only learn $n$ quanta. In order to cover as many samples as possible, the model should learn the top $n$ quanta. Assuming that each quantum reduces the loss of its corresponding samples from $b$ to $a$, we can estimate the model's average loss as
\begin{equation}L = a \sum_{k=1}^n p_k + b \sum_{k=n+1}^{\infty} p_k\end{equation}
The top $n$ quanta have already been learned, so the loss on this portion of samples is $a$; the remaining quanta have not been learned, so their loss is $b$. This assumption may look a bit strong — it might be more reasonable to make $a,b$ a function of $k$ — but the result is already representative enough (see the appendix of the original paper). For the expression above, we can carry out an asymptotic estimate:
\begin{equation}\begin{aligned} L =&\, a \sum_{k=1}^{\infty} p_k + (b - a) \sum_{k=n+1}^{\infty} p_k \\ =&\, a + (b - a) \sum_{k=n+1}^{\infty} \frac{k^{-\gamma-1}}{Z_{\gamma}} \\ \sim&\, a + (b - a) \int_n^{\infty} \frac{k^{-\gamma-1}}{Z_{\gamma}} dk \\ =&\, a + \frac{b - a}{\gamma Z_{\gamma}} n^{-\gamma} \\ \end{aligned}\end{equation}
This shows that the model's capability (i.e., the loss function) relates to the number of capability quanta $n$ via a power law of the form $n^{-\gamma}$. Clearly, $a$ here represents the minimum value of the loss function; if $a=0$, then $L\sim \mathcal{O}(n^{-\gamma})$. In what follows we will assume $a=0$.
Scaling Laws
In the basic result above, $n$ — the number of capability quanta the model has learned — has so far just been an abstract concept. Next, we will connect it to variables that commonly appear in models.
Parameter count: Suppose the model has $N$ parameters, and assume that on average it takes $C$ parameters to learn one capability quantum (with $C$ assumed to be a constant). Then clearly $n\propto N$, and
\begin{equation}L\sim \mathcal{O}(N^{-\gamma})\end{equation}
Data volume: Suppose the total number of samples in the training set is $D$. Since we assumed different quanta are responsible for handling samples of different difficulty, it is natural to assume that the number of samples handled by quantum $1$ is $Dp_1$, the number handled by quantum $2$ is $Dp_2$, the number handled by quantum $3$ is $Dp_3$, and so on. We further assume that learning a quantum requires at least $\tau$ samples, so quanta with index $Dp_k < \tau$ cannot be learned. From $\tau=D p_n$ we can solve for $n\propto D^{1/(\gamma + 1)}$, and substituting gives
\begin{equation}L\sim \mathcal{O}(D^{-\gamma/(\gamma + 1)})\end{equation}
Training volume: Suppose the model's parameter count and the training set's sample count are both unbounded, so the number of quanta the model has learned, $n$, depends only on the number of training steps $S$. Assuming the batch size is $B$, then on average, the number of samples used in learning quantum $1$ is $Bp_1$, for quantum $2$ it is $Bp_2$, for quantum $3$ it is $Bp_3$, and so on. Again assuming that learning a quantum requires at least $\tau$ samples, then after $S$ training steps, quantum $n$ has been trained on a total of $SBp_n$ samples. From $\tau=SB p_n$ we can solve for $n\propto S^{1/(\gamma + 1)}$, and substituting gives
\begin{equation}L\sim \mathcal{O}(S^{-\gamma/(\gamma + 1)})\end{equation}
We can see that although all these results are power laws, since $\gamma > \gamma/(\gamma + 1) \in (0, 1)$, it is clear that parameter count has a noticeably greater effect on model capability than the others.
Emergent Phenomena
Some readers may wonder: can the capability-quantization hypothesis be used to explain the emergence phenomenon in large models?
To some extent, yes. Earlier we assumed the perfect model has infinitely many capability quanta. If we change this "infinite" to "finite," then by increasing the parameter count, the model always has a chance of eventually covering all the capability quanta and reaching the theoretically optimal, perfect model — this is emergence. Alternatively, one could say the perfect model still has infinitely many capability quanta, but humans' "resolution" for perceiving intelligence only covers finitely many quanta (humans themselves may not be perfect either). So once a large model has learned enough capability quanta, from the human perspective it appears to have "emerged" as suddenly perfect.
Summary
This post introduced how to derive the model's Scaling Law from a quantization hypothesis — specifically, the asymptotic relationship between the model's loss function and parameter count, data volume, and training volume — and briefly analyzed its possible connection to the phenomenon of emergence.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.