Is β1=β2 the Optimal Hyperparameter for the Adam Optimizer?
I recently came across the paper Why Adam Works Better with β1=β2: The Missing Gradient Scale Invariance Principle, which, as the title suggests, claims that Adam performs better when $\beta_1=\beta_2$. A colleague pointed out that last year's paper In Search of Adam's Secret Sauce makes the same argument. Coincidentally, The Effect of Mini-Batch Noise on the Implicit Bias of Adam, which just came out yesterday, has similar findings too.
\begin{equation}\text{Adam}\color{skyblue}{\text{W}}:=\left\{\begin{aligned} &\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + \left(1 - \beta_1\right) \boldsymbol{g}_t\\ &\boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + \left(1 - \beta_2\right) \boldsymbol{g}_t^2\\ &\hat{\boldsymbol{m}}_t = \boldsymbol{m}_t\left/\left(1 - \beta_1^t\right)\right.\\ &\hat{\boldsymbol{v}}_t = \boldsymbol{v}_t\left/\left(1 - \beta_2^t\right)\right.\\ &\boldsymbol{u}_t =\hat{\boldsymbol{m}}_t\left/\left(\sqrt{\hat{\boldsymbol{v}}_t} + \epsilon\right)\right.\\ &\boldsymbol{\theta}_t = \boldsymbol{\theta}_{t-1} - \eta_t (\boldsymbol{u}_t \color{skyblue}{ + \lambda_t \boldsymbol{\theta}_{t-1}}) \end{aligned}\right.\end{equation}
So many papers converging on $\beta_1=\beta_2$ — what theoretical benefit does it actually offer? In this post, let's work through the relevant derivations. more
Online Estimation
In chronological order, let's start with In Search of Adam's Secret Sauce. The storyline of this paper seems to be: through experiments, the authors found that Adam's optimum under the constraint $\beta_1=\beta_2$ is very close to its unconstrained optimum, and they then tried to build a theoretical explanation for this — when $\beta_1=\beta_2=\beta$, $\hat{\boldsymbol{m}}_t$ and $\hat{\boldsymbol{v}}_t$ can be viewed as online estimates of the first and second moments of the gradient. Specifically, expanding $\hat{\boldsymbol{m}}_t$ and $\hat{\boldsymbol{v}}_t$, we get
\begin{equation}\hat{\boldsymbol{m}}_t = \frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} \boldsymbol{g}_k,\qquad \hat{\boldsymbol{v}}_t = \frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} \boldsymbol{g}_k^2\end{equation}
It is easy to show that the sum of coefficients $\frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k}$ is always equal to 1, so both are a form of weighted average of $\boldsymbol{g}_t$ and $\boldsymbol{g}_t^2$ respectively, and thus carry the meaning of a first moment and a second moment. Going further, we have
\begin{equation}\begin{aligned} \hat{\boldsymbol{v}}_t =&\, \frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} (\hat{\boldsymbol{m}}_t + \boldsymbol{g}_k - \hat{\boldsymbol{m}}_t)^2 \\ =&\, \underbrace{\frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} \hat{\boldsymbol{m}}_t^2}_{\hat{\boldsymbol{m}}_t^2} + \frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} (\boldsymbol{g}_k - \hat{\boldsymbol{m}}_t)^2 + \underbrace{\frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} 2\hat{\boldsymbol{m}}_t (\boldsymbol{g}_k - \hat{\boldsymbol{m}}_t)}_{\boldsymbol{0}} \\ =&\, \hat{\boldsymbol{m}}_t^2 + \frac{1-\beta}{1-\beta^t}\sum_{k=1}^t \beta^{t-k} (\boldsymbol{g}_k - \hat{\boldsymbol{m}}_t)^2 \\ \end{aligned}\end{equation}
The last term clearly has the form of a variance, so let's denote it $\hat{\boldsymbol{\sigma}}_t^2$, i.e., $\hat{\boldsymbol{v}}_t=\hat{\boldsymbol{m}}_t^2+\hat{\boldsymbol{\sigma}}_t^2$, which is exactly the relation between the second moment, the mean, and the variance. Note that all the additions, subtractions, multiplications, divisions, and powers of vectors involved in the above operations are element-wise: applied component by component, with the output still being a vector.
Signal-to-Noise Awareness
Under this new notation, Adam's update can be written as (for simplicity assume $\epsilon=0$):
\begin{equation}\newcommand{sign}{\mathop{\text{sign}}}\boldsymbol{u}_t = \frac{\hat{\boldsymbol{m}}_t}{\sqrt{\hat{\boldsymbol{v}}_t}} = \frac{\hat{\boldsymbol{m}}_t}{\sqrt{\hat{\boldsymbol{m}}_t^2+\hat{\boldsymbol{\sigma}}_t^2}} = \frac{\sign(\hat{\boldsymbol{m}}_t)}{\sqrt{1 +\hat{\boldsymbol{\sigma}}_t^2/\hat{\boldsymbol{m}}_t^2}}\end{equation}
This form of the update has some nice properties. Most obviously, each of its components is bounded, confined to $[-1, 1]$, so we don't need to worry about the update blowing up. Second, $\hat{\boldsymbol{\sigma}}_t^2/\hat{\boldsymbol{m}}_t^2$ has exactly the form of an inverse signal-to-noise ratio, so it can also be understood as "signal-to-noise-ratio-aware steepest descent."
Based on our derivation in Steepest Descent on Manifolds: 1. SGD + Hypersphere, $\sign(\hat{\boldsymbol{m}}_t)$ can be viewed as the solution to the following optimization problem:
\begin{equation}\max_{\boldsymbol{u}} \langle\hat{\boldsymbol{m}}_t,\boldsymbol{u}\rangle\qquad \text{s.t.}\qquad \Vert\boldsymbol{u}\Vert_{\infty} = 1\end{equation}
where $\Vert\boldsymbol{u}\Vert_{\infty} = 1$ means that the components of $\boldsymbol{u}$ have an absolute value of at most 1. If we treat $\hat{\boldsymbol{m}}_t$ as the more accurate gradient, then $\sign(\hat{\boldsymbol{m}}_t)$ is the steepest descent direction under the infinity norm. But this bound is currently static, and it's reasonable to think that if the gradient fluctuates little (a high signal-to-noise ratio), the region is relatively flat and the update can be enlarged accordingly, whereas if it fluctuates a lot, the update should be shrunk. So constructing a dynamic bound $\frac{1}{\sqrt{1 +\hat{\boldsymbol{\sigma}}_t^2/\hat{\boldsymbol{m}}_t^2}}$ for each component based on the signal-to-noise ratio better captures the effect of adaptive learning, and in that case the steepest descent direction becomes exactly
\begin{equation}\max_{\boldsymbol{u}} \langle\hat{\boldsymbol{m}}_t,\boldsymbol{u}\rangle\quad \text{s.t.}\quad |\boldsymbol{u}| \leq \frac{1}{\sqrt{1 +\hat{\boldsymbol{\sigma}}_t^2/\hat{\boldsymbol{m}}_t^2}} \qquad\Rightarrow\qquad \boldsymbol{u}^* = \frac{\sign(\hat{\boldsymbol{m}}_t)}{\sqrt{1 +\hat{\boldsymbol{\sigma}}_t^2/\hat{\boldsymbol{m}}_t^2}}\end{equation}
Here, the absolute value $||$ and the "less than or equal to" $\leq$ are both element-wise.
First-Order Expansion
Now let's turn to Why Adam Works Better with β1=β2: The Missing Gradient Scale Invariance Principle. It treats Adam as a continuous-time ODE. However, for mini-batch optimization, gradient noise is not negligible; continuizing into an SDE would be reasonable, but an ODE is simply not rigorous enough, so I feel the starting point of this paper is already quite forced.
Following the original paper's line of thought, I've made some adjustments to the proof process. Let's write each $\boldsymbol{g}_k$ in $\hat{\boldsymbol{v}}_t$ as $\hat{\boldsymbol{m}}_t + (\boldsymbol{g}_k - \hat{\boldsymbol{m}}_t)$, treat $\boldsymbol{g}_k - \hat{\boldsymbol{m}}_t$ as a small quantity, and do a first-order expansion to get
\begin{equation}\begin{aligned} \hat{\boldsymbol{v}}_t =&\, \frac{1-\beta_2}{1-\beta_2^t}\sum_{k=1}^t \beta_2^{t-k} (\hat{\boldsymbol{m}}_t + \boldsymbol{g}_k - \hat{\boldsymbol{m}}_t)^2 \\ \approx &\, \frac{1-\beta_2}{1-\beta_2^t}\sum_{k=1}^t \beta_2^{t-k} \hat{\boldsymbol{m}}_t^2 + \frac{1-\beta_2}{1-\beta_2^t}\sum_{k=1}^t \beta_2^{t-k} 2\hat{\boldsymbol{m}}_t (\boldsymbol{g}_k - \hat{\boldsymbol{m}}_t) \\ \approx &\, \hat{\boldsymbol{m}}_t^2 + 2\hat{\boldsymbol{m}}_t \left(\frac{1-\beta_2}{1-\beta_2^t}\sum_{k=1}^t \beta_2^{t-k} \boldsymbol{g}_k - \hat{\boldsymbol{m}}_t\right) \\ \end{aligned}\end{equation}
We then want $\hat{\boldsymbol{v}}_t$ to be as close as possible to $\hat{\boldsymbol{m}}_t^2$, so the first-order term should vanish, from which we obtain $\beta_2 = \beta_1$. Why do we want $\hat{\boldsymbol{v}}_t$ to be close to $\hat{\boldsymbol{m}}_t^2$? Precisely so that the update $\boldsymbol{u}_t = \hat{\boldsymbol{m}}_t/\sqrt{\hat{\boldsymbol{v}}_t}$ can be closer to $\sign(\hat{\boldsymbol{m}}_t)$, because $\sign$ is bounded and thus more resistant to perturbations arising from scale changes in $\boldsymbol{g}_t$, improving training stability.
It should be pointed out that the proof here is drastically simplified compared to the original, but it captures the core idea and corrects it. The original paper first continuizes into an ODE, which itself is factually questionable, then applies some non-rigorous approximations, and ends up with an expansion in $\boldsymbol{u}_t = \sign(\boldsymbol{g}_t)(1 + \cdots)$ — a precision that is arguably less reliable than what we obtain here by expanding directly at $\hat{\boldsymbol{m}}_t$.
A Dual Optimization Problem
There's a common thread running through these two papers: making $\boldsymbol{u}_t$ bounded in order to improve training stability. This prompted me to think about it in reverse: given $\beta_1$, what value should $\beta_2$ take to make $|\boldsymbol{u}_t|$ as small as possible? Since $\boldsymbol{u}_t = \hat{\boldsymbol{m}}_t/\sqrt{\hat{\boldsymbol{v}}_t}$, intuitively we'd want $\hat{\boldsymbol{v}}_t$ to be as large as possible, which leads to the following dual optimization problem:
\begin{equation}\max_{\beta_2} \min_{\boldsymbol{g}_1,\cdots,\boldsymbol{g}_t}\underbrace{\frac{1-\beta_2}{1-\beta_2^t}\sum_{k=1}^t \beta_2^{t-k} \boldsymbol{g}_k^2}_{\hat{\boldsymbol{v}}_t},\qquad \text{s.t.}\qquad \frac{1-\beta_1}{1-\beta_1^t}\sum_{k=1}^t \beta_1^{t-k} \boldsymbol{g}_k = \hat{\boldsymbol{m}}_t\end{equation}
The notation here isn't perfectly rigorous — just understand it component-wise. Taking $\min$ over $\boldsymbol{g}_1,\cdots,\boldsymbol{g}_t$ in the objective cannot be dropped: it represents the requirement that the chosen $\beta_2$ perform as well as possible for any gradient sequence. This optimization problem looks complicated, but it isn't really — we can solve it layer by layer using the Cauchy–Schwarz inequality. First, solving the inner minimization problem, we have
\begin{equation}\sum_{k=1}^t \frac{p_k^2}{q_k}\times \sum_{k=1}^t q_k \boldsymbol{g}_k^2 \geq \left(\sum_{k=1}^t p_k \boldsymbol{g}_k\right)^2 = \hat{\boldsymbol{m}}_t^2\end{equation}
where $p_k = \frac{1-\beta_1}{1-\beta_1^t}\beta_1^{t-k},q_k = \frac{1-\beta_2}{1-\beta_2^t}\beta_2^{t-k}$. This means the result of the inner minimization is $\frac{\hat{\boldsymbol{m}}{}_t^2}{\sum_{k=1}^t p_k^2/q_k}$, and to maximize it we need to minimize $\sum_{k=1}^t p_k^2/q_k$. Using Cauchy–Schwarz again,
\begin{equation}\sum_{k=1}^t p_k^2/q_k = \sum_{k=1}^t q_k \times \sum_{k=1}^t p_k^2/q_k \geq \left(\sum_{k=1}^t p_k\right)^2 = 1\end{equation}
with equality achieved at $q_k=p_k$, which means $\beta_2 = \beta_1$. In short, taking $\beta_2=\beta_1$ gives the best training stability, guaranteeing $\hat{\boldsymbol{v}}_t\geq \hat{\boldsymbol{m}}_t^2$ — and this time we derived it directly from the dual optimization objective, rather than deducing it after already knowing $\beta_1=\beta_2$.
Related Work
Some readers may wonder: Adam's original paper, Adam: A Method for Stochastic Optimization, recommends a default value of $\beta_1=0.9,\beta_2=0.999$, and our earlier analysis in Adaptive Learning-Rate Optimizers Viewed Through Hessian Approximation also pointed to $\beta_2 > \beta_1$ — don't these conclusions contradict each other?
Not really — they're just different conclusions under different overall settings. In the early days, what we trained were, relatively speaking, small models with small batch sizes, so the noise at each step was fairly large. Setting a larger $\beta_2$ makes $\hat{\boldsymbol{v}}_t$ change more slowly, making Adam behave more like SGD, where updates across nearby steps can be linearly superimposed to further reduce noise. The paper The Effect of Mini-Batch Noise on the Implicit Bias of Adam reports a similar finding: with small batch sizes $\beta_1 < \beta_2$, while with large batch sizes the two become closer to each other.
As for Adaptive Learning-Rate Optimizers Viewed Through Hessian Approximation, that was the result of an approximate analysis near an idealized optimum. But nowadays, the models we train keep getting larger, and so do batch sizes. We find that after training a model for a month, the loss might already look decent — but if we have enough compute and data, training for another month or two would push the loss down even further. In other words, whether we train for one month or two, the model is still far from the true optimum.
Under this new regime, the core logic of training has shifted to "stability first, speed second": stability is the overriding prerequisite — we believe that as long as we can train stably and continuously, results will keep improving — and $\beta_1=\beta_2$, by keeping the update bounded, precisely satisfies this expectation of "stability." Indeed, in the LLM era, Adam's default hyperparameters have gradually shifted from $\beta_1=0.9,\beta_2=0.999$ to $\beta_1=0.9,\beta_2=0.95$, moving closer to $\beta_1=\beta_2$, which corroborates our reasoning.
Summary
In this post, we analyzed the $\beta_1,\beta_2$ hyperparameters of the Adam optimizer and showed, from a stability perspective, that $\beta_1=\beta_2$ is generally a better choice, which can be understood as steepest descent under signal-to-noise-ratio awareness.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.