An Identity for ReLU/GeLU/Swish

ReLU/GeLU/Swish的一个恒等式 苏剑林 · 2025-08-16 · Mathematics analysis neural network identity

Today let's have something lighter—based on an identity I stumbled upon over the past couple of days. It's actually quite simple, but at first glance it feels somewhat unexpected, so it's worth recording here.

Basic Result

We know that $\newcommand{relu}{\mathop{\text{relu}}}\relu(x) = \max(x, 0)$, from which it's easy to prove the following identity:

\begin{equation}x = \relu(x) - \relu(-x)\end{equation}

If $x$ is a vector, then the equation above becomes more intuitive: $\relu(x)$ extracts the positive components of $x$, $- \relu(-x)$ extracts the negative components of $x$, and adding them together recovers the original vector.

A General Conclusion

The next question is: does a similar identity hold for activation functions like GeLU] and Swish]? At first glance it doesn't seem to hold, but in fact it does! We even have a more general result:

Let $\phi(x)$ be any odd function, $f(x)=\frac{1}{2}(\phi(x) + 1)x$, then the following always holds:
\begin{equation}x = f(x) - f(-x)\end{equation}

Proving this result is also quite easy, so I won't go into the details here. For Swish we have $\phi(x) = \tanh(\frac{x}{2})$, and for GeLU we have $\phi(x)=\mathop{\text{erf}}(\frac{x}{\sqrt{2}})$; both are odd functions, so the same identity holds for them.

Some Thoughts on Its Significance

Written in matrix form, the identity above becomes

\begin{equation}x = f(x) - f(-x) = f(x[1, -1])\begin{bmatrix}1 \\ -1\end{bmatrix}\end{equation}

This shows that when using ReLU, GeLU, Swish, or similar activation functions, a two-layer network has the capacity to degenerate into a one-layer network. In other words, such networks can adaptively adjust their effective depth—which works in much the same spirit as ResNet does. This might be part of the reason why these activation functions tend to outperform traditional choices like Tanh or Sigmoid.