On the Design of Activation Functions in Neural Networks

浅谈神经网络中激活函数的设计 苏剑林 · 2017-10-26 · Machine Learning neural network

The activation function is the source of nonlinearity in a neural network: if you remove these functions, the entire network reduces to a sequence of linear operations, and the composition of linear operations is still a linear operation—the final effect would be no more than a single-layer linear model.

So what are the common activation functions? Or rather, what guiding principles exist for choosing an activation function? Can any arbitrary nonlinear function serve as an activation function?

Here we're discussing activation functions for the hidden layers, not the output layer. The final output usually has a specific activation function that can't be changed arbitrarily—for instance, binary classification typically uses sigmoid activation, multi-class classification typically uses softmax activation, and so on. Hidden layers, by contrast, offer much more freedom of choice.

Even Floating-Point Error Will Do!

In theory, any nonlinear function could potentially serve as an activation function. A very compelling example is OpenAI's recent success in using floating-point error as an activation function. For the details, see OpenAI's blog post:

https://blog.openai.com/nonlinear-computation-in-linear-networks/

or the write-up from Synced (机器之心):

https://mp.weixin.qq.com/s/PBRzS4Ol_Zst35XKrEpxdwmore

That said, different activation functions come with different training costs. Although OpenAI's exploration showed that even floating-point error can serve as an activation function, this operation is non-differentiable, so they had to resort to "evolution strategies" to train the model. "Evolution strategies," as the name suggests, refers to time- and resource-hungry algorithms like genetic algorithms.

The Precedent Set by ReLU

So if we add the requirement of differentiability, so that gradient descent can be used to train the model, are we then in the clear? Not really. In the early days of neural networks, the sigmoid function was the standard choice of activation:

$$\begin{equation}\text{sigmoid}(x)=\sigma(x)=\frac{1}{1+e^{-x}}\end{equation}$$

This function's defining trait is that it approaches 0 on the left and 1 on the right—both ends saturate. See the figure below:

sigmoid

Precisely because of this, the derivative approaches 0 at both ends. Since we optimize with gradient descent, and the update magnitude is proportional to the gradient, a derivative near zero means each update is tiny, making optimization difficult. This is especially true once there are many layers: due to the chain rule, the update magnitude ends up proportional to the gradient raised to the $n$-th power, which makes optimization even harder. This is why early neural networks couldn't be made very deep.

A landmark activation function was ReLU, whose definition is very simple:

$$\begin{equation}\text{relu}(x)=\max(x,0)\end{equation}$$

Its graph looks like this:

relu

This is a piecewise linear function, and clearly its derivative is 1 on the positive half and 0 on the negative half. So across the whole real line, half the space is non-saturating. Sigmoid, by comparison, saturates almost everywhere (the saturated region's share of the domain tends to 1, where "saturated" means the derivative is very close to 0).

ReLU is piecewise linear, so its nonlinearity is quite weak, which is why networks using it generally need to be made very deep. But this actually suits our needs well, because—given comparable results—depth tends to matter more than width, and deeper models tend to generalize better. So ever since ReLU came along, all sorts of very deep models have been proposed. A landmark event here was the VGG model and its success on ImageNet; I won't go into the subsequent developments in detail.

The Better Swish

Despite ReLU's impressive track record, some people felt that having half the input space saturate was still a significant shortcoming, and proposed variants such as Leaky ReLU and PReLU—these changes are all fairly similar in spirit.

A few days ago, the Google Brain team proposed a new activation function called Swish; you can read about it here:

http://mp.weixin.qq.com/s/JticD0itOWH7Aq7ye1yzvg

It's defined as

$$\begin{equation}\text{swish}(x)=x\cdot\sigma(x)=\frac{x}{1+e^{-x}}\end{equation}$$

and its graph looks like this:

Swish

The team's test results show that this function outperforms ReLU across many models.

Visually, Swish looks a lot like ReLU—the main difference is in the region on the negative side near zero. With the benefit of hindsight, I'll admit that I myself had thought about this kind of activation function before, because it's similar to Facebook's GLU activation function, defined as

$$\begin{equation}(\boldsymbol{W}_1\boldsymbol{x}+\boldsymbol{b}_1)\otimes \sigma(\boldsymbol{W}_2\boldsymbol{x}+\boldsymbol{b}_2)\end{equation}$$

That is, you train two separate sets of parameters, activate one of them with sigmoid, and multiply it with the other. Here $\sigma(\boldsymbol{W}_2\boldsymbol{x}+\boldsymbol{b}_2)$ is called the "gate," which is exactly what the G in GLU stands for. Swish, then, is essentially the special case where both sets of parameters are taken to be the same—so you only need to train one set.

Ideas for Improvement

Swish has stirred up some controversy—some people felt Google Brain was making a fuss over a small thing, that tweaking an activation function is the kind of work a small team could do, and that a big outfit like Google Brain should be aiming for loftier goals. Regardless, Google Brain ran a lot of experiments, and the results consistently show Swish outperforming ReLU. So it's worth asking: what's the underlying reason for this?

The analysis below is purely the blogger's own speculation, with no theoretical or experimental proof to back it up at this point—read with appropriate skepticism. I think an important reason Swish beats ReLU has to do with initialization.

Swish is non-saturating near the origin; it only saturates in the negative region far from the origin, whereas ReLU already has half its space saturated right around the origin. When we train models, we typically use uniform or normal initialization for the parameters, and in either case the mean is generally 0. That means half of the initialized parameters already sit in ReLU's saturated region, so half the parameters go unused right from the start. This is made worse by techniques like batch normalization, which push outputs to automatically approximate a zero-mean normal distribution—so again, roughly half the parameters end up in ReLU's saturated region. Swish is a bit better in this regard, since it still has some non-saturating region on the negative side, giving better utilization of the parameters.

As I mentioned, I myself had thought about this kind of activation function before, but never pursued it seriously—partly because it didn't feel elegant or clean to me; I even found it a bit ugly. But seeing how good Swish's experimental results were, I started wondering whether there might be a similar, more aesthetically pleasing activation function. I came up with this one:

$$\begin{equation}x\cdot\min(1,e^x)\end{equation}$$

Its graph is:

An activation function I came up with myself

It actually looks pretty similar to Swish. The basic idea was: keep $x$ on the positive side, and on the negative side, come up with a function that first decreases and then increases while still tending to 0—I thought of $xe^{-x}$, and with a bit of tuning I arrived at this function. In some of my own models, it actually performed slightly better than Swish (on my QA model). Of course, I've only done a little bit of experimentation—I don't have the time or compute to run a thorough comparison.

Here's a comparison with Swish, shown in orange.

Comparison with Swish (orange)

One caveat: if you want to use this function, don't implement it directly in this form, because computing $e^x$ can overflow. Here's a version that won't overflow:

$$\begin{equation}\max(x, x\cdot e^{-|x|})\end{equation}$$

or, written using ReLU:

$$\begin{equation}x + \text{relu}(x\cdot e^{-|x|}-x)\end{equation}$$

Could it be that all the good activation functions end up looking like the checkmark (√) on our homework?