Deconstructing Scaling Laws: A Triad of Optimization, Architecture, and Data
When training a large neural network, the final outcome is influenced by an enormous number of factors: swap the optimizer, swap the architecture, or swap the training set, and the results can be dramatically different. In engineering practice, we affectionately call the empirical process of tuning these factors "alchemy." But how can we elevate this from mere experience to a genuine regularity — a more precise, quantitative description of the relationships involved? If we could crack this problem, our alchemy would rest on much firmer ground.
"Scaling Law" is an attempt to answer this question in a relatively quantitative way. Since OpenAI's foundational 2020 work (Kaplan Law), Scaling Laws have become one of the most reliable empirical regularities in deep learning — usable for predicting model performance, guiding hyperparameter choices, and judging whether a given modification actually helps. Accordingly, a number of works have tried to dig deeper and offer more fundamental interpretations of Scaling Laws.
This post shares some of my own understanding of Scaling Laws. more
Preliminaries
Later on we will repeatedly encounter problems of the form "minimize a combination of power laws under some constraint," so let's first prepare two basic results.
The mismatched-power inequality
First, an inequality we'll use extensively. Let $a,b,p,q,x > 0$; then
\begin{equation}a x^p + b x^{-q} \geq (p+q)\left(\frac{a^{q}b^{p}}{p^{p}q^{q}}\right)^{\frac{1}{p+q}}\end{equation}
with equality holding when
\begin{equation}x=\left(\frac{bq}{ap}\right)^{\frac{1}{p+q}}\end{equation}
This is a generalization of the simple inequality $x + x^{-1} \geq 2$, and can be proved either by differentiation or via the weighted AM-GM inequality. Let's demonstrate the latter. The weighted AM-GM inequality states $w_1 x_1 + w_2 x_2 \geq (w_1 + w_2)(x_1^{w_1} x_2^{w_2})^{\frac{1}{w_1+w_2}}$, so we have
\begin{equation}a x^p + b x^{-q} = q \cdot \frac{a x^p}{q} + p \cdot \frac{b x^{-q}}{p} \geq (q+p)\left[\left(\frac{a x^p}{q}\right)^q \left(\frac{b x^{-q}}{p}\right)^p\right]^{\frac{1}{p+q}} = (p+q)\left(\frac{a^{q}b^{p}}{p^{p}q^{q}}\right)^{\frac{1}{p+q}}\end{equation}
with equality holding when $\frac{a x^p}{q} = \frac{b x^{-q}}{p}$, i.e., $x=\left(\frac{bq}{ap}\right)^{\frac{1}{p+q}}$. For convenience we'll refer to this result as the "mismatched-power inequality," whose main feature is that it involves two power functions with opposite-signed exponents. Interestingly, both the minimizing point and the minimum value themselves remain power laws — this is one of the cornerstones of the derivations that follow.
Optimal allocation ratio
The mismatched-power inequality fixes a product ($(x^p)^q(x^{-q})^p=1$) and finds the minimum of a sum of two power-law terms. In some scenarios (e.g., finding the optimal parameter allocation), we instead need to fix their "sum" and minimize. Specifically, let $a,b,p,q,x,y > 0$; we want to solve
\begin{equation}\min_{x,y} a x^{-p} + b y^{-q}\qquad\text{s.t.}\qquad x+y=1\end{equation}
Unfortunately, this problem has no elementary closed-form solution, but numerically solving it poses no difficulty. It's not hard to see that a minimum must exist: substituting $y=1-x$ and differentiating gives $bq(1-x)^{-q-1}-apx^{-p-1}$; setting this to zero yields
\begin{equation}\frac{x^{p+1}}{(1-x)^{q+1}} = \frac{ap}{bq}\end{equation}
The left-hand side is clearly monotonically increasing in $x$ over $(0, 1)$, and as $x\to 1$, the left side tends to $\infty$, while as $x\to 0$, the left side tends to 0. By the intermediate value theorem, a unique solution exists within $(0, 1)$, which can be found directly via bisection.
Conceptual Framework
The process of training a model can be formalized as:
Given data $\mathcal{D}$, architecture $\mathcal{A}$, and optimizer $\mathcal{O}$, minimize the loss function $L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O})$.
Here $\mathcal{E}$ denotes some idealized distribution — you can think of it as an unimaginably huge test set — while any training set $\mathcal{D}$ we can actually construct is a subset of it, or a sample drawn from it. $L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O})$ then denotes the loss value the model can achieve on the idealized distribution $\mathcal{E}$ under these conditions.
Threefold decomposition
Consider the following decomposition:
\begin{equation}\begin{aligned} L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O}) &\,= \underbrace{L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\mathcal{O})}_{\text{data}} \\ &\,\qquad + \underbrace{L(\mathcal{D}|\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\infty)}_{\text{optimize}} \\ &\,\qquad\qquad + \underbrace{L(\mathcal{D}|\mathcal{A},\infty) - L(\mathcal{D}|\infty,\infty)}_{\text{architecture}} \\ &\,\qquad\qquad\qquad + L(\mathcal{D}|\infty,\infty) \\ \end{aligned}\end{equation}
This decomposition might look like it's complicating things, but in fact it breaks the gap between the current training state and the ideal target into three successive, cascading steps. Under the usual "more is better" assumption, each bracketed term is non-negative, so this rewrites the total gap as a sum of three individually interpretable distances, decoupling as much as possible the influence of each variable on the loss function, so that we can more reasonably infer the dependencies among them.
Let's now explain the meaning of each term in turn.
Data error
The first level of decomposition is
\begin{equation}L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O}) = \Big[L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\mathcal{O})\Big] + L(\mathcal{D}|\mathcal{A},\mathcal{O})\end{equation}
where $L(\mathcal{D}|\mathcal{A},\mathcal{O})$ denotes the model's loss on the training set $\mathcal{D}$, given architecture $\mathcal{A}$ and optimizer $\mathcal{O}$.
Note that, by definition, $L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O})$ denotes the model's loss under the idealized distribution $\mathcal{E}$ — this is our ultimate target. However, $\mathcal{E}$ is untouchable during training; we can only work with the training set $\mathcal{D}$, obtaining the training loss $L(\mathcal{D}|\mathcal{A},\mathcal{O})$, and must then find some way to characterize the gap between $L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\mathcal{O})$ and the target via other means.
This term is usually called the "generalization error," and its key drivers are data-related — the quantity, quality, and diversity of the data, for instance. The architecture $\mathcal{A}$ and optimizer $\mathcal{O}$ may also affect the generalization error; which variables we need to consider depends on our analytical goal.
Optimization error
The second level of decomposition is
\begin{equation}L(\mathcal{D}|\mathcal{A},\mathcal{O}) = \Big[L(\mathcal{D}|\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\infty)\Big] + L(\mathcal{D}|\mathcal{A},\infty)\end{equation}
where $L(\mathcal{D}|\mathcal{A},\infty)$ denotes the ideal loss achievable on the training set when optimization is pushed to its limit — say, with a perfect super-optimizer, or with the number of training steps and hyperparameter tuning attempts pushed to infinity.
So $L(\mathcal{D}|\mathcal{A},\infty)$ represents the ceiling imposed by the optimizer, while $L(\mathcal{D}|\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\infty)$ measures the gap between the practical optimizer and this ceiling — it captures whether the optimizer is good enough: is the learning rate well-chosen, are there enough training steps, is the batch size large enough to stabilize the gradient, and so on.
Architecture error
The third level of decomposition is
\begin{equation}L(\mathcal{D}|\mathcal{A},\infty) = \Big[L(\mathcal{D}|\mathcal{A},\infty) - L(\mathcal{D}|\infty,\infty)\Big] + L(\mathcal{D}|\infty,\infty)\end{equation}
where $L(\mathcal{D}|\infty,\infty)$ denotes the most ideal loss achievable on the training set when both optimization and architecture are pushed to their limits — an arbitrarily excellent optimizer paired with an arbitrarily powerful model.
So $L(\mathcal{D}|\infty,\infty)$ is the theoretical limit determined by the dataset itself, while $L(\mathcal{D}|\mathcal{A},\infty) - L(\mathcal{D}|\infty,\infty)$ measures the performance gap between the practical model and this theoretical limit — it captures whether the architecture is good enough: is the parameter count sufficient, is the depth/width sufficient, is there still room to improve the residual connections, and so on.
Optimization
In this section we'll explore the optimization gap $F_{\text{opt}} = L(\mathcal{D}|\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\infty)$. Given the optimizer (e.g., Adam or Muon), we mainly care about the effects of three core hyperparameters: the learning rate $\eta$, the batch size $B$, and the number of training steps $T$. In principle, one could account for other parameters like momentum and other details too, but here we'll focus on these three.
Relationship analysis
First, a reasonable assumption is "the more training, the better the outcome," where "more training" carries two meanings: "more steps" and "a larger learning rate." Or, intuitively, we can think of $T\eta$ as the "distance traveled" by the model — the further it travels, the better the outcome — so we can guess that one contributing term is $\alpha_1 (T\eta)^{-\gamma_1}$.
On the other hand, training performance is also affected by noise, and a reasonable assumption is that more noise means worse performance. Noise comes from two sources: first, the batch size $B$ — the smaller it is, the greater the noise; second, the learning rate, which represents how "rough" the training trajectory is — a larger learning rate also means more noise. So we guess there's another term $\alpha_2 B^{-\gamma_2} + \alpha_3\eta^{\gamma_3}$.
Adding these two parts together gives
\begin{equation}F_{\text{opt}} \sim \alpha_1 (T\eta)^{-\gamma_1} + \alpha_2 B^{-\gamma_2} + \alpha_3\eta^{\gamma_3}\label{eq:optimizer-law}\end{equation}
We can also understand the above formula by splitting the training process into two stages: early in training, noise is a secondary factor, and the term $\alpha_1 (T\eta)^{-\gamma_1}$ drives the loss down quickly; later, noise gradually takes over and the model begins oscillating around the target, tracing out something like a spiral-descent trajectory.
This form matches the one used in 2503.12645 and 2603.15958. It's worth pointing out that these two works are not empirical curve-fits — they instead directly analyze the convergence properties of optimizers like SignSGD and Muon theoretically, arriving at $\gamma_1=\gamma_3=1,\gamma_2=1/2$. In the derivations below, we can plug in these values for a quick sanity check.
Optimal learning rate
Equation $\eqref{eq:optimizer-law}$ has 6 parameters; fitting it directly would require a huge number of data points and would be very costly, and it would also be quite prone to overfitting. To address this, we can leverage the optimal-parameter assumption to simplify the form further.
First, by the mismatched-power inequality, we can find the optimal learning rate that minimizes the right-hand side:
\begin{equation}\eta^* = \left(\frac{\gamma_1\alpha_1 T^{-\gamma_1}}{\gamma_3\alpha_3}\right)^{\frac{1}{\gamma_1+\gamma_3}} \sim T^{-\frac{\gamma_1}{\gamma_1+\gamma_3}}\end{equation}
with the corresponding minimum value being
\begin{equation}F_{\text{opt}}^* = \underbrace{(\gamma_1+\gamma_3)\left(\frac{(\alpha_1 T^{-\gamma_1})^{\gamma_3} \alpha_3^{\gamma_1}}{\gamma_1^{\gamma_1} \gamma_3^{\gamma_3}}\right)^{\frac{1}{\gamma_1+\gamma_3}}}_{\sim T^{-\frac{\gamma_1\gamma_3}{\gamma_1+\gamma_3}}} + \alpha_2 B^{-\gamma_2}\end{equation}
This tells us two things: (1) there exists some $0 < c < 1$ such that the optimal learning rate is inversely proportional to $T^c$; (2) assuming we can always find the optimal learning rate for every configuration, the asymptotic behavior of the optimization error simplifies to the form $\tilde{\alpha}_1 T^{-\tilde{\gamma}_1} + \alpha_2 B^{-\gamma_2}$, cutting the number of parameters down to 4 — exactly the decoupled form proposed by 2607.01487. Meanwhile 2605.09154, starting from a noisy quadratic model, likewise arrives at the conclusion that $B,T$ should be modeled separately.
Optimal batch size
Now, starting from the optimal-learning-rate assumption, let the optimization error be
\begin{equation}F_{\text{opt}} \sim \tilde{\alpha}_1 T^{-\tilde{\gamma}_1} + \alpha_2 B^{-\gamma_2}\end{equation}
Write $K = B T$, denoting the number of samples seen during training — note we haven't restricted things to a single epoch, so it's possible some samples get learned more than once. If we fix $K$, the right-hand side becomes $\tilde{\alpha}_1 (B / K)^{\tilde{\gamma}_1} + \alpha_2 B^{-\gamma_2}$, and applying the mismatched-power inequality again gives the minimum value
\begin{equation}F_{\text{opt}}^* = (\tilde{\gamma}_1 + \gamma_2) \left(\frac{(\tilde{\alpha}_1 K^{-\tilde{\gamma}_1})^{\gamma_2} \alpha_2^{\tilde{\gamma}_1}}{\tilde{\gamma}_1^{\tilde{\gamma}_1} \gamma_2^{\gamma_2}}\right)^{\frac{1}{\tilde{\gamma}_1+\gamma_2}} \;\sim\; K^{-\frac{\tilde{\gamma}_1 \gamma_2}{\tilde{\gamma}_1+\gamma_2}}\end{equation}
with equality at
\begin{equation}B^* = \left(\frac{\alpha_2 \gamma_2 K^{\tilde{\gamma}_1}}{\tilde{\alpha}_1 \tilde{\gamma}_1}\right)^{\frac{1}{\tilde{\gamma}_1 + \gamma_2}} \sim K^{\frac{\tilde{\gamma}_1}{\tilde{\gamma}_1+\gamma_2}}\end{equation}
This again gives us two conclusions: given a total number of training samples $K$, the optimal batch size $B^*$ is proportional to $K^c$, where $c\in(0, 1)$; and under the optimal batch size, the Scaling Law simplifies to $\hat{\alpha}_1 K^{-\hat{\gamma}_1}$, which is exactly the classic Scaling Law form.
Summary
Let's summarize: the general Scaling Law for an optimizer is $\alpha_1 (T\eta)^{-\gamma_1} + \alpha_2 B^{-\gamma_2} + \alpha_3\eta^{\gamma_3}$. If we assume we're always running at the optimal learning rate, it simplifies to $\tilde{\alpha}_1 T^{-\tilde{\gamma}_1} + \alpha_2 B^{-\gamma_2}$; and if we further fix the number of training samples $K$ and assume we can always find the optimal batch size, it simplifies further to $\hat{\alpha}_1 K^{-\hat{\gamma}_1}$.
For the optimal parameters: the optimal batch size is proportional to some power (no greater than 1) of $K$, which matches Step Law; the optimal learning rate is inversely proportional to some power (no greater than 1) of $T$, and converting via $T^*=K/B^*$, the optimal learning rate is also inversely proportional to some power (no greater than 1) of $K$, which matches Microsoft Law but contradicts Step Law.
Plugging in the theoretical values mentioned earlier, $\gamma_1=\gamma_3=1,\gamma_2=1/2$, we get $\tilde{\gamma}_1 = 1/2$, and hence $B^*\sim K^{1/2}$, which is fairly close to the $B^*\sim K^{0.571}$ given by Step Law. Also we have $\eta^* \sim T^{-1/2}$, and converting via $T^*=K/B^*$ gives $\eta^* \sim K^{-1/4}$, not too far from Microsoft Law's $\eta^* \sim K^{-0.32}$. Finally, we get $F_{\text{opt}}^*\sim K^{-1/4}$, which is also close to the $\sim K^{-0.28}$ given by Chinchilla Law.
Architecture
Next we turn to the model gap $F_{\text{arch}} = L(\mathcal{D}|\mathcal{A},\infty) - L(\mathcal{D}|\infty,\infty)$, which discusses purely the contribution of the architecture $\mathcal{A}$ to the loss function. Classic variables here include parameter count $N$, width $W$, and depth $H$, while the introduction of these architectural variables in turn feeds back into the behavior of the optimization gap. Let's try to sort all this out.
Model parameters
Holding the overall architecture fixed, the main variable of the model is its parameter count $N$. Assuming a larger parameter count is always better, it's reasonable to posit
\begin{equation}F_{\text{arch}} \sim \alpha_4 N^{-\gamma_4} \label{eq:arch-law-N}\end{equation}
This is the most basic Scaling Law assumption about parameter count, matching Kaplan Law, Chinchilla Law, and others, where Kaplan Law gives $\gamma_4=0.076$ while Chinchilla Law's fitted result is $\gamma_4 = 0.34$; it's now generally believed that Chinchilla Law is more accurate at large training scales.
One point of contention here is whether the parameter count $N$ should include the embeddings. The mainstream practice is to exclude them, but this can introduce sizable bias at small scale, which is likely part of why the results of Kaplan Law and Chinchilla Law differ (in the era of Kaplan's experiments, training scales were generally small). The paper 2406.12907 analyzes this in detail. For a more accurate treatment of the embedding's contribution, see the later section "The Layer of Memory."
Rather than compressing the whole architecture into a single parameter count $N$, we can be more fine-grained — for instance, separating width $W$ and depth $H$, to study whether the model should be "tall and thin" or "short and fat":
\begin{equation}F_{\text{arch}}\sim \alpha_W W^{-\gamma_W} + \alpha_H H^{-\gamma_H}\end{equation}
Given that the parameter count is roughly $N\sim W^2 H$, we can, under fixed parameter count, solve for the optimal width, depth, and the corresponding $F_{\text{arch}}^*$:
\begin{equation}W^* \sim N^{\frac{\gamma_H}{\gamma_W+2\gamma_H}},\qquad H^* \sim N^{\frac{\gamma_W}{\gamma_W+2\gamma_H}},\qquad F_{\text{arch}}^* \sim N^{-\frac{\gamma_W\gamma_H}{\gamma_W+2\gamma_H}}\end{equation}
The paper 2606.25008, working from theory (2505.10465, 2602.05970), proposes $\gamma_W=\gamma_H=1$, and substituting gives
\begin{equation}W^* \sim N^{1/3},\qquad H^* \sim N^{1/3},\qquad F_{\text{arch}}^* \sim N^{-1/3}\end{equation}
The resulting $F_{\text{arch}}^* \sim N^{-1/3}$ is still quite close to Chinchilla Law.
Optimization behavior
The parameter count also influences the optimization process. In equation $\eqref{eq:optimizer-law}$, the three coefficients $\alpha_1,\alpha_2,\alpha_3$ were treated as constants under the assumption of a fixed architecture $\mathcal{A}$. Now that we've introduced the parameter count $N$, $\alpha_1,\alpha_2,\alpha_3$ naturally becomes a function of $N$ as well.
Again let's understand this in two parts: on the one hand, a larger parameter count means a more capable model whose loss falls faster, so we replace $\alpha_1$ with $\alpha_1 N^{-\gamma_5}$; on the other hand, a larger parameter count means a more complex model with more noise, so we replace $\alpha_2,\alpha_3$ with $\alpha_2 N^{\gamma_6}$ and $\alpha_3 N^{\gamma_7}$ respectively, giving
\begin{equation}F_{\text{opt}} \sim \alpha_1 N^{-\gamma_5}(T\eta)^{-\gamma_1} + \alpha_2 N^{\gamma_6} B^{-\gamma_2} + \alpha_3 N^{\gamma_7}\eta^{\gamma_3}\end{equation}
A natural question arises here: why do we only consider $\alpha_1,\alpha_2,\alpha_3$ varying with $N$, and not the exponent $\gamma_1,\gamma_2,\gamma_3$? We'll address this question in the section "On Power Laws." Repeating the calculation from the "Optimization" section, we get
\begin{gather}\eta^* \sim T^{-\frac{\gamma_1}{\gamma_1+\gamma_3}} \cdot N^{-\frac{\gamma_5+\gamma_7}{\gamma_1+\gamma_3}} \sim K^{-\frac{\gamma_1\gamma_2}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \cdot N^{\frac{\gamma_1\gamma_6-\gamma_7(\gamma_1+\gamma_2)-\gamma_5\gamma_2}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \\ B^* \sim K^{\frac{\gamma_1\gamma_3}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \cdot N^{\frac{\gamma_6(\gamma_1+\gamma_3)+\gamma_5\gamma_3-\gamma_7\gamma_1}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \\ F_{\text{opt}}^* \sim K^{-\frac{\gamma_1\gamma_2\gamma_3}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \cdot N^{\frac{\gamma_1\gamma_2\gamma_7+\gamma_1\gamma_3\gamma_6-\gamma_2\gamma_3\gamma_5}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \end{gather}
If we accept the results of Kaplan Law, Chinchilla Law, and Step Law, then $B^*$ and $F_{\text{opt}}^*$ should be independent of $N$. Setting the exponents of these two $N$ terms to zero, we can solve
\begin{equation}\gamma_3\gamma_5 = \gamma_1\gamma_7, \qquad \gamma_6 = 0\end{equation}
Substituting all this back in, we find only one new parameter $\gamma_7$ appears:
\begin{gather}F_{\text{opt}}\sim \alpha_1 N^{-\frac{\gamma_1\gamma_7}{\gamma_3}}(T\eta)^{-\gamma_1} + \alpha_2 B^{-\gamma_2} + \alpha_3 N^{\gamma_7}\eta^{\gamma_3} \\ \eta^* \sim T^{-\frac{\gamma_1}{\gamma_1+\gamma_3}} \cdot N^{-\frac{\gamma_7}{\gamma_3}} \sim K^{-\frac{\gamma_1\gamma_2}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \cdot N^{-\frac{\gamma_7}{\gamma_3}} \\ B^* \sim K^{\frac{\gamma_1\gamma_3}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \\ F_{\text{opt}}^* \sim K^{-\frac{\gamma_1\gamma_2\gamma_3}{\gamma_1\gamma_3+\gamma_2(\gamma_1+\gamma_3)}} \end{gather}
Interestingly, the form of $\eta^*$ turns out to match Microsoft Law exactly, being negatively correlated with both $K$ and $N$ — and this is not a trivial fact, since we only assumed that $B^*$ and $F_{\text{opt}}^*$ are independent of $N$, with no assumption made about $\eta^*$. Finally, plugging in the theoretical values $\gamma_1=\gamma_3=1,\gamma_2=1/2$ gives
\begin{equation} \eta^* \sim K^{-1/4} N^{-\gamma_7},\qquad B^* \sim K^{1/2},\qquad F_{\text{opt}}^* \sim K^{-1/4} \end{equation}
As for $\gamma_7$, Microsoft Law gives $0.23$, while Step Law gives $0.713$. Given that these two Laws depend on $K$ in completely opposite ways, it's natural that they differ noticeably in this exponent. Theory leans somewhat more toward Microsoft Law; from certain convex-optimization results, $N^{\gamma_7}$ is related to the standard deviation of the gradient over all parameters, which suggests it should lie somewhere between $0\sim 0.5$, and combined with Microsoft Law, we might guess $1/4$.
Given a fixed compute budget
Combining $F_{\text{opt}}^*$ and $F_{\text{arch}}$, we have
\begin{equation}F_{\text{opt}}^* + F_{\text{arch}} \sim \hat{\alpha}_1 K^{-\hat{\gamma}_1} + \alpha_4 N^{-\gamma_4}\end{equation}
For dense models, the compute cost of each step is basically proportional to the parameter count $N$, and $K$ is the number of samples seen during training, which is likewise proportional to the training compute. So the total compute cost consumed during training is $C\sim NK$, and the proportionality constant for the standard architecture is roughly $6$, comprising $2NK$ for the forward pass and $4NK$ for the backward pass.
In practical training, compute is usually limited, and we want to achieve maximum "intelligence" under a fixed compute budget $C$, which requires finding the optimal $K^*$ and $N^*$ under the constraint $NK \sim C$. Substituting $K \sim C/N$ into the equation above (absorbing constant factors into the coefficient), we get
\begin{equation}F_{\text{opt}}^* + F_{\text{arch}} \sim \hat{\alpha}_1 C^{-\hat{\gamma}_1} N^{\hat{\gamma}_1} + \alpha_4 N^{-\gamma_4}\end{equation}
This is once again in the shape of a mismatched-power inequality, so applying the formula directly gives
\begin{equation}N^* \sim C^{\frac{\hat{\gamma}_1}{\hat{\gamma}_1+\gamma_4}}, \qquad K^* \sim C^{\frac{\gamma_4}{\hat{\gamma}_1+\gamma_4}}, \qquad F^* \sim C^{-\frac{\hat{\gamma}_1\gamma_4}{\hat{\gamma}_1+\gamma_4}}\end{equation}
If we plug in the theoretical values $\hat{\gamma}_1=1/4$ and $\gamma_4=1/3$ from before, then
\begin{equation}N^* \sim C^{3/7}, \qquad K^* \sim C^{4/7}, \qquad F^* \sim C^{-1/7}\end{equation}
This is quite close to the core conclusion of Chinchilla Law: the optimal model size and the optimal amount of data should scale roughly proportionally (the paper's fitted result is $N^* \sim C^{0.46}, \qquad K^* \sim C^{0.54}$). Finally, substituting the expressions for $N^*$ and $K^*$ into $\eta^*\sim K^{-1/4} N^{-\gamma_7}$ and $B^*\sim K^{1/2}$ from the previous section, we get
\begin{equation}\eta^*\sim C^{-(1+3\gamma_7)/7}, \qquad B^*\sim C^{2/7}\end{equation}
Here $B^*\sim C^{2/7}$ is actually not too far from the $B^*\sim C^{0.3271}$ of DeepSeek Law, but DeepSeek Law gives $\eta^*\sim C^{-0.1250}$, whereas here even plugging in $\gamma_7=1/4$ only yields $\eta^*\sim C^{-1/4}$, which is still quite far off. So it looks like different works are fairly consistent on the optimal batch size, but diverge quite a bit on the optimal learning rate — probably related to the specific optimization setup, learning rate schedule, and so on.
Sparse architectures
Earlier we said "compute is roughly proportional to parameter count $N$" — this holds for dense models. Suppose the model's core operation is a linear layer, where an input of size $a\times b$ is multiplied by parameters of size $b\times c$; the compute is $\mathcal{O}(abc)$ and the parameter count is $bc$, both proportional to $bc$, so parameter count is essentially compute. But in recent years people have also devoted effort to architectures that decouple parameter count from compute, such as MoE.
MoE is naturally the dominant architecture today, without question — almost every open-source large model is an MoE. An important new hyperparameter it introduces is the sparsity $S$, which can be defined as the ratio of total parameters to activated parameters, or the ratio of total experts to activated experts. In theory, the theoretical compute of MoE depends roughly only on the activated parameter count, meaning that increasing sparsity $S$ theoretically does not increase compute, while the loss can still decrease — so increasing sparsity is always worthwhile.
A simple way to incorporate sparsity into the Scaling Law is:
\begin{equation}F_{\text{arch}} \sim \alpha_4 N_{act}^{-\gamma_{act}} N_{total}^{-\gamma_{total}} = \alpha_4 N_{act}^{-(\gamma_{act}+\gamma_{total})} S^{-\gamma_{total}}\label{eq:arch-law-N-act-total}\end{equation}
where $N_{act},N_{total}$ are the activated parameter count and total parameter count respectively, and $S=N_{total}/N_{act}$; a similar form also appears in 2501.12370. Looking at equation $\eqref{eq:arch-law-N}$, this amounts to assuming that $S$ only affects the coefficient $\alpha_4$ in a power-law manner, without affecting the exponent $\gamma_4$. From the above equation, we can also derive the concept of an "effective parameter count":
\begin{equation}N_{eff} = N_{act}^{\frac{\gamma_{act}}{\gamma_{act}+\gamma_{total}}} N_{total}^{\frac{\gamma_{total}}{\gamma_{act}+\gamma_{total}}},\qquad F_{\text{arch}} \sim \alpha_4 N_{act}^{-\gamma_{act}} N_{total}^{-\gamma_{total}} = \alpha_4 N_{eff}^{-(\gamma_{act}+\gamma_{total})}\end{equation}
That is, an MoE model with activated parameter count $N_{act}$ and total parameter count $N_{total}$ is equivalent to a dense model with parameter count $N_{eff}$; a commonly used empirical formula in the community is $N_{eff}=\sqrt{N_{act}N_{total}}$ (reference). This concept can be traced back to the Effective Parameter Count proposed in 2202.01169, which was later extended by Ling Law into an "efficiency leverage," whose scaling behavior was also explored there.
However, since compute is only proportional to the activated parameter count $N_{act}$, this means we could keep $N_{eff}$ fixed while letting $N_{act}\to 0$ — in other words, driving compute down to nearly zero while keeping performance unchanged, which does not seem realistic. Therefore, it is reasonable to suspect that $F_{\text{arch}}$ should include an additional penalty term on $N_{act}$ to guarantee a certain amount of activated parameters:
\begin{equation}F_{\text{arch}} \sim \alpha_4 N_{act}^{-\gamma_{act}} N_{total}^{-\gamma_{total}} + \alpha_8 N_{act}^{-\gamma_8}\label{eq:arch-law-moe}\end{equation}
Other work modeling sparsity includes 2309.08520, 2501.12370, 2502.05172, etc., each proposing a different form of Scaling Law. Besides sparsity, the granularity of experts (first modeled by 2402.07871) and shared experts also have some effect on performance, and 2509.23678 combines all these factors into a very elaborate form for empirical fitting.
Of course, "increasing sparsity is always worthwhile" is only true in theory; in practice one also has to consider routing overhead, inference efficiency, and other issues — increasing sparsity is not entirely free, and requires joint design of algorithms and infrastructure.
Memory layers
Besides MoE, other ways to decouple parameter count from compute include sparse Memory layers, classic examples being PKM and UltraMem, while newer ones like Over-Encoding and Engram also fall into this category.
In fact, this line of work can also be seen as another extreme of MoE: their "experts" are simplified into trainable vectors with no computation at all, and then a trainable router (pointer) or N-gram hash is used to select which "expert" to activate. From this perspective, their Scaling Law should be similar to MoE's — for example, a dense model equipped with several memory layers should also exhibit a Scaling Law of the form of equation $\eqref{eq:arch-law-moe}$.
If these two different kinds of sparse designs — MoE and Memory — are used together, what should the Scaling Law look like? Let's first define some notation $N_{act}, N_{moe}, N_{mem}, N_{total}$, representing respectively the "activated parameter count (counting only parameters that produce compute, i.e., the dense part plus activated experts)", "the total parameter count excluding Memory", "the total parameter count excluding non-activated experts", and "the total parameter count." They satisfy the identity
\begin{equation}N_{moe} + N_{mem} = N_{total} + N_{act}\label{eq:N-id}\end{equation}
If we guess that MoE and Memory play complementary roles, their Scaling Law might be additive, i.e.,
\begin{equation}F_{\text{arch}} \sim \alpha_{4a} N_{act}^{-\gamma_{act}} N_{moe}^{-\gamma_{moe}} + \alpha_{4b} N_{act}^{-\gamma_{act}} N_{mem}^{-\gamma_{mem}} + \alpha_8 N_{act}^{-\gamma_8}\end{equation}
This also raises a new optimization problem: if we must fix the activated parameter count $N_{act}$ (the compute bottleneck) and the total parameter count $N_{total}$ (the memory bottleneck), how should we allocate parameters between MoE and Memory? According to the identity $\eqref{eq:N-id}$, $N_{moe} + N_{mem}$ is then a constant, and we need to minimize the above under this constraint. Setting $N_{moe} = \lambda(N_{total} + N_{act})$ and substituting into the equation, the problem becomes a sum-constrained minimization problem as introduced in the section "Optimal allocation ratio," which can be solved numerically.
In other words, it is not optimal to allocate all parameters to either MoE or Memory alone — there exists an optimal ratio, which is consistent with the findings of Engram.
Summary
Similar to the optimization section, this section also starts from an empirical power-law assumption and then mainly uses the mismatched-power inequality for optimization.
The simplest variable for quantifying the model is the total parameter count $N$; in more detail, one can consider the width $W$ and depth $H$ separately. Changes in the model also induce changes in the optimization error, so we also briefly discussed the joint effect of various optimizer parameters and parameter count. Under the dense-model assumption, the parameter count itself also represents the compute, and the optimizer's number of training steps $T$ and batch size $B$ are also positively correlated with compute — so we can further explore, under a fixed total budget $C$, how to determine the optimal parameter count and the corresponding optimizer parameters.
If we refine things further, there are even more variables in the model architecture: MoE architectures distinguish between "activated parameters" and "total parameters," and Memory designs such as Engram are similar; and when they occur together, an optimal-allocation problem for parameters arises as well. Residual-connection improvements such as MHC and AttnRes haven't even been mentioned yet, and so on. In short, there are a great many variables to consider on the model side, and we can only give a brief overview here.
Data
Finally, we turn to the data gap $F_{\text{data}} = L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O}) - L(\mathcal{D}|\mathcal{A},\mathcal{O})$. By definition, this term requires us to focus on the performance on the ideal distribution $\mathcal{E}$, but we've also said that the ideal distribution is theoretically unreachable — so how should we measure it?
In fact, there's no good solution: we can only pick a dataset that hasn't been trained on and that we believe is sufficiently representative, and use the test loss computed on it as an approximation of $L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O})$. So, similar to benchmarks, constructing the test set is especially important, as it represents how accurately we understand and describe our ideal objective.
Data size
The core hyperparameter for data is the size of the training set $D$; we believe that the larger the training set, the better the generalization performance (i.e., "more exposure, more knowledge"), so it contributes a term $\alpha_9 D^{-\gamma_9}$.
Since the data at each step is sampled uniformly at random from the training set, $K/D$ represents the average number of times each sample is trained on — that is, the number of training epochs. We believe that the more severe the multi-epoch repetition, the worse the generalization performance, so it contributes a term $\alpha_{10} (K/D)^{\gamma_{10}}$, giving a basic form of
\begin{equation}F_{\text{data}} \sim \alpha_9 D^{-\gamma_9} + \alpha_{10} (K/D)^{\gamma_{10}}\end{equation}
Most Scaling Law works, under the single-epoch assumption, treat $K$ and $D$ as the same quantity; here we distinguish them in order to be able to account for multi-epoch effects.
The recent paper Prescriptive Scaling Laws for Data Constrained Training also introduces a similar power law, but replaces $K/D$ with $K/D-1$, while also taking into account the effect of model parameter count, roughly assuming that the larger the model, the more prone it is to overfitting, and hence the less it should be trained for multiple epochs — so $\alpha_{10}$ is made positively correlated with $N$. These modifications can also be adjusted as needed.
On the other hand, papers such as Scaling Data-Constrained Language Models introduce the concept of "value decay," converting the amount of data after multi-epoch repetition into an "effective data amount," thereby correcting the Scaling Law. But the author feels this term is only needed because $K$ and $D$ were not distinguished; having already distinguished $K$ and $D$, we only need to directly penalize the overfitting risk brought about by multi-epoch training.
Optimal number of epochs
Combining the optimization error and the data error (fixing $D$, $N$, and assuming training always uses the optimal hyperparameters), we get:
\begin{equation}F_{\text{opt}}^* + F_{\text{data}} \sim \hat{\alpha}_1 K^{-\hat{\gamma}_1} + \alpha_{10} (K/D)^{\gamma_{10}} \end{equation}
This clearly shows: the more we train, the smaller the optimization error, but the more severe the data repetition, the larger the generalization error. So, once again applying the mismatched-power inequality, we can find the optimal value of $K$:
\begin{equation}K^* = \left(\frac{\hat{\alpha}_1 \hat{\gamma}_1}{\alpha_{10} \gamma_{10}} D^{\gamma_{10}}\right)^{\frac{1}{\hat{\gamma}_1+\gamma_{10}}} \sim D^{\frac{\gamma_{10}}{\hat{\gamma}_1+\gamma_{10}}} \qquad \frac{K^*}{D} \sim D^{-\frac{\hat{\gamma}_1}{\hat{\gamma}_1+\gamma_{10}}}\end{equation}
This gives us a scaling law regarding multi-epoch training, with the conclusion: the less data there is, the more epochs one should actually train for; the more data there is, the smaller the optimal number of epochs — a conclusion that is exactly the opposite of 2511.13421. One possible refinement is to generalize $\alpha_{10} (K/D)^{\gamma_{10}}$ to $\alpha_{10} K^{\gamma_{10}} D^{-\gamma_{11}}$, in which case we have
\begin{equation}K^* = \left(\frac{\hat{\alpha}_1 \hat{\gamma}_1}{\alpha_{10} \gamma_{10}} D^{\gamma_{11}}\right)^{\frac{1}{\hat{\gamma}_1+\gamma_{10}}} \sim D^{\frac{\gamma_{11}}{\hat{\gamma}_1+\gamma_{10}}} \qquad \frac{K^*}{D} \sim D^{\frac{\gamma_{11}-\hat{\gamma}_1-\gamma_{10}}{\hat{\gamma}_1+\gamma_{10}}}\end{equation}
With this, the optimal number of epochs could either increase or decrease as the amount of data increases.
But even after switching to $\alpha_{10} K^{\gamma_{10}} D^{-\gamma_{11}}$, there are still some unreasonable aspects here — for example, when $K\to\infty$, it still tends to infinity, whereas empirically, even if we keep training indefinitely, the test loss should not diverge to infinity. That said, if we only fit within a small range (assuming the number of multi-epoch repetitions cannot be too large), the power-law assumption may still give practically usable results.
Further thoughts
Besides the amount of data $D$ as the core parameter, a lot of work has made fine-grained distinctions in the composition of the data, such as domain mixture ratios (2403.16952, 2507.09404, 2603.19149, 2605.12715, 2606.08167), quality level (2510.03313), modality mixture ratios (2607.22043), and so on. These works are quite varied, and it's hard to distill them into a unified form, so we won't go through them one by one.
Compared to optimization and architecture, the Scaling Laws on the data side really do give a sense of being "messy" and "vague." The reason isn't hard to understand: the variables for the optimizer ($\eta,B,T$) and the architecture ($N,W,H,S$) are all fairly well-defined numbers, whereas on the data side, aside from the data volume $D$, which can be quantified fairly precisely, dimensions like specialized domain, quality, and modality don't really have clear boundaries to begin with, nor can they be accurately captured by a single scalar.
Moreover, from the notation $L(\mathcal{E}|\mathcal{D},\mathcal{A},\mathcal{O})$ we can see that $\mathcal{D},\mathcal{A},\mathcal{O}$ are all conditions of it — meaning that changes in the optimizer variables and architecture variables will, in principle, also affect the data-side Scaling Law. With all these variables interacting with one another, it becomes very difficult to study the dependence patterns of data cleanly, in an isolated setting.
What's even more troublesome is that, on closer reflection, the very proposition of a "Data Scaling Law" is itself puzzling: in order to measure performance fairly, we need to first prepare a sufficiently representative test set, and then "pretend" not to know what the test set looks like, while trying to study the "Scaling Law" on the training data so that performance on the test set is as good as possible — no matter how you look at it, this procedure is more than a little strange.
Summary
In this section we gave a brief introduction to Scaling Laws for the data gap, mainly introducing the data-size parameter $D$, taking into account the positive effect of increasing the amount of data as well as the overfitting risk brought about by multiple epochs, and combining this with the optimization error to derive a result for the optimal number of epochs. Overall, though, there is still a lot on the data side of Scaling Laws that remains puzzling and demands deeper thought.
On Power Laws
Up to this point, all the variation patterns we've dealt with have been assumed to take the form of sums or products of "power laws," from which we then derived optimal choices via the "unequal-power inequality." Looking back now, two questions are worth pondering: 1) Why should we assume the variation pattern is a power law in the first place? 2) When we introduce additional conditions, why do we only consider changes to the power-law coefficient, and not changes to the exponent?
The Question of Power Laws
Why power laws? Many researchers have tried to propose more "essential" explanations, but in the author's view, a lot of these attempts merely swap one assumption for another without any real change—see, for instance, Deriving Scaling Laws from a Quantization Hypothesis.
The author's view is that the most direct explanation is this: once we've established that a variable's dependence is monotonically decreasing, and we only care about its asymptotic behavior, there really aren't many function families to choose from—basically power functions and exponential functions. Exponential functions decay too fast—in the language of distributions, they're "short-tailed"—which means that a modest investment of some resource quickly hits a ceiling in returns, and this doesn't match our intuitive sense of how the real world works. Power functions, by contrast, are "long-tailed": they decay more slowly, and better capture the phenomenon of "sustained investment yielding sustained improvement."
Put more philosophically: if this world were dominated by exponential functions, it would be a rather boring place—everything would quickly hit a ceiling, and further investment would quickly become pointless. It's precisely because power laws dominate that the story of scaling up becomes so compelling.
Another way to look at it: power laws are equivalent to "scale invariance": $f(\lambda x)=\lambda^{-\gamma} f(x)$—the function looks the same at any scale, and the very meaning of the word "scaling" is a pattern that holds across scales. Among elementary functions, it seems only the power law can play this role. From a practical standpoint, a power law is a straight line in log-log coordinates, which makes it easy to fit and visualize; and the fact that the "unequal-power inequality" exists, and that its minimizer and minimum value are themselves still power laws, are important reasons why it can serve as an "empirical law."
Of course, the power law isn't the only answer—in the "Data" section we saw a paper that used an exponential function to describe the decaying value of multiple epochs. All we can really say is that the power law is the basic pattern worth trying first in most cases; if we run into a problem that's particularly hard to fit this way, we can consider making adjustments. Or, put differently: if the range over which we need to extrapolate isn't very large, then what matters most is probably the monotonicity of the function, and the differences between various functional forms may not actually matter that much.
The Question of Coefficients
The second question first came up in the "Architecture" section: we assumed that a change in conditions could, at most, affect the coefficient of a power law—for instance, a change in architecture only affects the coefficient of the optimization-error power law, and a change in MoE sparsity only affects the coefficient of the architecture-error power law, while the exponent stays fixed. What's the reasoning behind this? Is there some deeper principle at play here?
First, there's a pragmatic mathematical reason: if the exponent were also allowed to vary with conditions, the power-law form itself would break down—the result would no longer be a power law, the unequal-power inequality would no longer apply, and the whole framework would lose its simplicity and tractability. Letting only the coefficient vary is the smallest generalization that preserves the form, and it's the assumption worth trying first.
Second, there's a physical analogy: in statistical physics, critical exponents at a phase transition are universal—different materials within the same universality class share the same set of exponents, and material-specific details only change the non-universal prefactor. Similarly, we can understand the exponent in a Scaling Law as reflecting the intrinsic "difficulty" of the problem itself—determined by the data distribution and the task—while the coefficient reflects the "engineering quality" of the solution, which changes as the optimizer or architecture improves. But this is, in essence, doing a better job on the same problem, so what should change is the coefficient in front of the power law, not the exponent.
Turning this around: if some finite engineering improvement really could change the exponent, then, asymptotically, the relative advantage it produces would grow without bound as scale increases—a finite investment yielding an unbounded relative return, which amounts to conjuring exponential-level progress out of thin air. This is generally unrealistic, unless the improvement actually changes the nature of the problem itself—but then it's no longer "an engineering improvement on the same problem," it's a different problem altogether.
Of course, this is mostly a conjecture on the author's part, based on considerations of simplicity and self-consistency—consider it an open question, and discussion is welcome.
Conclusion
Having read through this post, some readers might feel like "it seems like everything was said, and yet nothing was really said." We haven't proven any theorems—we've simply written down a three-way decomposition, jotted down a bunch of empirically motivated power-law assumptions, and then done some basic analysis using optimization methods.
What we've tried to do, in this way, is to identify the commonalities across various Scaling Law results and clarify how they interact with one another. Fortunately, this post has indeed yielded some heuristic results, though given the limits of the author's understanding, most of the content could only be introduced at a fairly superficial level—especially the "Data" section, where the author's own understanding remains quite shallow.
Looking back at the whole derivation process, there's a certain flavor of "dimensional analysis" from physics to it: it's not as rigorous as a first-principles derivation—rather, we guess the power-law form of each term based on intuition and monotonicity, use this to derive patterns in the optimal solution, and finally compare against classical results, simplifying or correcting as needed.
I hope this perspective proves useful to readers in understanding and applying Scaling Laws.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.