【Note to self】On Dropout

【备忘】谈谈dropout 苏剑林 · 2017-08-08 · Machine Learning deep learning

Actually this is just a memo for myself...

Dropout is an effective measure against overfitting in deep learning. Of course, in terms of the underlying idea, dropout isn't restricted to deep learning either — it can also be used in traditional machine learning methods. It's just that within the neural network framework of deep learning, dropout feels more natural.

What does it do

How does dropout actually operate? Generally speaking, given an input tensor $x$, dropout sets some of its elements to zero, and then applies a scale transformation to the zeroed-out result. Concretely, taking Keras's Dropout(0.6)(x) as an example, this is actually equivalent to the following thing done in numpy:

import numpy as np

x = np.random.random((10,100)) #模拟一个batch_size=10、维度为100的输入
def Dropout(x, drop_proba):
    return x*np.random.choice(
                              [0,1], 
                              x.shape,  
                              p=[drop_proba,1-drop_proba]
                             )/(1.-drop_proba)

print Dropout(x, 0.6)

That is to say, 60% of the elements are set to 0, and the remaining 40% are scaled up to 1/40% = 2.5 times their original value. One thing worth noting: in Keras, the 0.6 in Dropout(0.6)(x) denotes the drop ratio, whereas in TensorFlow, the 0.6 in tf.nn.dropout(x, 0.6) denotes the keep ratio. Which meaning applies depends on the specific framework (though, of course, if the dropout ratio happens to be 0.5, you don't need to worry about it ^_^).

What's it good for

We generally understand dropout as "a low-cost ensembling strategy," and that's correct. Here's roughly how the process can be understood.

After the zeroing operation above, we can think of the zeroed-out part as having been discarded — some information is lost. But even though information is lost, life must go on — well, actually, training must go on — so the model is forced to fit the target using only the remaining information. Since each round of dropout is random, the model can't afford to rely too heavily on any particular set of nodes. So overall, the effect is this: each time, the model is forced to learn from a small subset of features, and the subset that gets used changes every time, which means every feature should end up contributing to the model's prediction (rather than the model leaning heavily on a subset of features, which would lead to overfitting).

Finally, at prediction time, dropout is turned off, so the result is equivalent to an average over all the local feature subsets (this time, finally, all the information gets used). In theory this improves performance, and overfitting becomes less severe too (because the risk is spread evenly across every feature rather than concentrated on just a few).

Getting more flexible

Sometimes we want to impose certain constraints on how dropout is applied. For example, we might want all samples within the same batch to be dropped out in the same way, rather than having the first sample drop only its first node while the second sample drops only its second node. Or, when doing text classification with an LSTM, we might want to drop out entire word embeddings at a time — that is, for each word, either the whole embedding vector is dropped or the whole thing is kept, rather than dropping out only some of the components within a single word embedding. Or again, when classifying RGB images with a CNN, we might want to drop out by channel — for each image, drop out any one of the R, G, B channels entirely (somewhat like a color transform, or an RGB perturbation), rather than dropping out individual pixels across the image (which is more like adding noise to the image).

To achieve these kinds of behavior, we need the noise_shape parameter of dropout. This parameter is available both in Keras's Dropout layer and in tf.nn.dropout (these are the only two frameworks I know), and it means the same thing in both. However, this parameter is rarely discussed online, and even when it is mentioned, the explanations tend to be vague. Let's use tf.nn.dropout(x, 0.5, noise_shape) as an example. First, noise_shape is a one-dimensional tensor — in plain terms, a one-dimensional array (which can be a list or tuple) — whose length must match x.shape. Moreover, each element in noise_shape must be either 1 or equal to the corresponding element in x.shape. For instance, if x.shape=(3,4,5), then noise_shape can only be one of the following 8 valid options:

(3,4,5), (1,4,5), (3,1,5), (3,4,1), (1,1,5), (1,4,1), (3,1,1), (1,1,1)

So what does each of these mean? Here's a way to think about it: whichever axis is set to 1, dropout is applied uniformly along that axis. For example, (3,4,5) is just ordinary dropout with no constraints at all; (1,4,5) means that every sample in the batch is dropped out in the exact same way (you can think of it as adding the same noise pattern to every sample); and if (3,4,5) represents (number of sentences, number of words per sentence, dimensionality of each word embedding), then (3,4,1) means dropping out by whole word embeddings (which you can think of as randomly skipping certain words).

For some readers, it might be more intuitive to see this expressed as numpy code, so here it is:

def Dropout(x, drop_proba, noise_shape):
    return x*np.random.choice(
                              [0,1], 
                              noise_shape, 
                              p=[drop_proba,1-drop_proba]
                             )/(1.-drop_proba)