Why is the default clipping norm for gradient clipping equal to 1?

为什么梯度裁剪的默认模长是1？ 苏剑林 · 2025-01-02 · Machine Learning optimization gradient learning rate optimizer

As we know, gradient clipping is a common technique for making model training more stable. The usual form of gradient clipping rescales the gradient based on the total norm across all parameters, and can be written as

\begin{equation}\text{clip}(\boldsymbol{g},\tau)=\left\{\begin{aligned}&\boldsymbol{g}, &\Vert\boldsymbol{g}\Vert\leq \tau \\ &\frac{\tau}{\Vert\boldsymbol{g}\Vert}\boldsymbol{g},&\Vert\boldsymbol{g}\Vert > \tau \end{aligned}\right.\end{equation}

In this way, $\text{clip}(\boldsymbol{g},\tau)$ keeps the same direction as $\boldsymbol{g}$, but its norm no longer exceeds $\tau$. Note that $\Vert\boldsymbol{g}\Vert$ here is the norm computed by treating the gradients of all the model's parameters together as a single vector — this is the so-called Global Gradient Norm.

I wonder if readers have noticed a small detail: whether a model has millions or tens of billions of parameters, the value chosen for $\tau$ is very often just 1. What does this mean? Is it simply people reusing a default value, or is there some deeper principle hiding behind it? More below the fold.

What it is

Some readers might think: the default value isn't necessarily the optimal value, so what's there to fuss about? True, $\tau=1$ is not necessarily the optimal choice, but it is the default choice for many models, and under this default choice the models perform reasonably well — which in turn suggests that $\tau=1$ has a kind of universal reasonableness.

What exactly does "reasonableness" mean here? Let's go back to the $\text{clip}$ operation. If $\Vert\boldsymbol{g}\Vert$ is always smaller than $\tau$, then $\text{clip}$ degenerates into the identity transform; if $\Vert\boldsymbol{g}\Vert$ is always larger than $\tau$, then $\text{clip}$ degenerates into L2 normalization. In other words, the reason $\text{clip}$ is set to $\text{clip}$ is precisely because $\tau$ creates a meaningful distinction: most values of $\Vert\boldsymbol{g}\Vert$ are smaller than $\tau$, and only a small fraction exceed $\tau$. This is what I mean by the "reasonableness" of $\tau$.

Of course one can find counterexamples, and quite a few of them at that — here I mainly want to emphasize the generality of this phenomenon and the universality of this default setting, so readers who like to nitpick needn't get too hung up on individual exceptions.

So, in our view, the universal reasonableness of $\tau=1$ means that regardless of how many parameters a model has, how it's initialized, or which loss function is used, its total gradient norm ends up, roughly, straddling $1$ as the boundary marking "abnormal values." This is truly a remarkable property — that was certainly my own reaction the first time I realized this.

Why

Why is this "coincidence" the case? My answer might be a bit surprising: because only under this condition is stable training of the model even possible.

Consider a loss function $\mathcal{L}(\boldsymbol{\theta})$, with the optimizer's update rule given by $\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\, \boldsymbol{u}_t$. Then the change in the loss function can be approximated as

\begin{equation}\Delta \mathcal{L} = \mathcal{L}(\boldsymbol{\theta}_{t+1}) - \mathcal{L}(\boldsymbol{\theta}_t) \approx (\boldsymbol{\theta}_{t+1} - \boldsymbol{\theta}_t)\cdot\nabla_{\boldsymbol{\theta}_t}\mathcal{L}(\boldsymbol{\theta}) = -\eta\, \boldsymbol{u}_t\cdot \boldsymbol{g}_t\end{equation}

Let's first consider plain SGD, for which $\boldsymbol{u}_t = \boldsymbol{g}_t$ and $\Delta \mathcal{L}=-\eta\Vert\boldsymbol{g}_t\Vert^2$, i.e., the change in the loss function is proportional to the square of the gradient norm. We know that whether in CV or NLP, pure SGD (without momentum) is a very inefficient optimizer; in the middle-to-late stages of training, for most tasks the average per-step decrease in loss is far smaller than the learning rate itself, i.e. $|\Delta \mathcal{L}| < \eta$, from which we can derive $\Vert\boldsymbol{g}_t\Vert < 1$. This shows that $\Vert\boldsymbol{g}_t\Vert < 1$ is the long-run behavior of a model that is converging normally.

Of course, in the early stages of training the model may well exhibit $\Vert\boldsymbol{g}_t\Vert > 1$ — that's normal — but it's rare to see $\Vert\boldsymbol{g}_t\Vert \gg 1$, or rather, a good initialization should avoid $\Vert\boldsymbol{g}_t\Vert \gg 1$ occurring; this is exactly the theoretical basis behind things like DeepNorm. The reasoning is similar: if the gradient norm is too large, then early learning becomes too "aggressive," causing premature convergence to a poor local solution. Another remedy is to shrink $\eta$, which likewise shrinks $|\Delta \mathcal{L}|$ — this is exactly why we typically use Warmup at the start of training.

By the way, for a good understanding of Warmup, readers can refer to the paper Optimal Linear Decay Learning Rate Schedules and Further Refinements, which I consider the most sensible analysis of Warmup available.

What to do about it

In short, since the change in the loss function is proportional to the square of the gradient norm, the requirement of stable training means the gradient norm cannot be too large, and in the long run it stays below 1. If, in the early stages, the gradient norm is noticeably larger than 1, the usual remedy is Warmup. Alternatively, one can consider a more general strategy: set another threshold $\mathcal{T}$, and clip $\eta$ based on the value of $\boldsymbol{u}_t\cdot \boldsymbol{g}_t$

\begin{equation}\eta_t = \left\{\begin{aligned}&\eta,& \boldsymbol{u}_t\cdot \boldsymbol{g}_t\leq \mathcal{T} \\ &\frac{\mathcal{T}}{\boldsymbol{u}_t\cdot \boldsymbol{g}_t}\eta,& \boldsymbol{u}_t\cdot \boldsymbol{g}_t > \mathcal{T} \end{aligned}\right.\end{equation}

This does away with the need for a separate Warmup schedule, and is more adaptive.

For optimizers like Adam, we can perform an approximate analysis using $\boldsymbol{u}_t=\text{sign}(\boldsymbol{g}_t)$, just as in How Should the Learning Rate Change as the Batch Size Increases?, in which case

\begin{equation}\Delta \mathcal{L} = -\eta\, \text{sign}(\boldsymbol{g}_t)\cdot \boldsymbol{g}_t = -\eta\, \Vert\boldsymbol{g}_t\Vert_1\end{equation}

Here $\Vert\Vert_1$ is the L1 norm, i.e., the sum of the absolute values of the components. Since the gradient components are almost all smaller than 1, we have $\Vert\boldsymbol{g}_t\Vert_1 \gg \Vert\boldsymbol{g}_t\Vert$ — so again, out of the need for stable training, Adam's learning rate typically needs to be noticeably smaller than SGD's learning rate. Furthermore, the equation above can also be rewritten as

\begin{equation}\Delta \mathcal{L} = -\eta\, \text{sign}(\boldsymbol{g}_t)\cdot \boldsymbol{g}_t = -\eta\, \sqrt{N}\Vert\boldsymbol{g}_t\Vert \cos(\text{sign}(\boldsymbol{g}_t), \boldsymbol{g}_t) \end{equation}

Here we assume $\boldsymbol{g}_t$ has no zero components, so $\Vert\text{sign}(\boldsymbol{g}_t)\Vert=\sqrt{N}$, where $N$ is the total number of model parameters. Empirically, $\Vert\boldsymbol{g}_t\Vert$ and $\cos(\text{sign}(\boldsymbol{g}_t), \boldsymbol{g}_t)$ are found to be roughly constant across different model scales, so in order to keep $\Delta \mathcal{L}$ unchanged, $\eta$ should scale inversely with $\sqrt{N}$ — meaning that if the parameter count increases by a factor of 4, the learning rate can be considered for halving.

The End

This post has offered some of my own observations and thoughts on the phenomenon that "the default norm for gradient clipping is 1."