From SamplePairing to mixup: A Magical Regularization Term

从SamplePairing到mixup：神奇的正则项 苏剑林 · 2018-07-07 · Machine Learning model

SamplePairing and mixup are two closely related image data augmentation techniques. They seem quite unreasonable, the operations involved are extremely simple, yet the results are remarkably good: across multiple image classification tasks, both have been shown to improve the accuracy of the final classification model.

Some readers might be puzzled by one question: why can such an unreasonable data augmentation method achieve such good results? This post aims to show that while they look like data augmentation methods, they are in fact a form of regularization for the model. As a classic line from Stephen Chow's film From Beijing with Love puts it:

On the surface it looks like a hair dryer, but it's actually a razor.

Data Augmentation

Let's start with data augmentation. Data augmentation refers to the practice of applying simple transformations to raw data, transformations that typically don't change the corresponding label, allowing us to "manufacture" more data from the data we already have. For example, if we take a photo of a dog and apply horizontal flipping, slight rotation, cropping, translation, and so on, we consider its category unchanged — it's still the same dog. In this way, from a single sample we can derive several samples, thereby increasing the size of the training set.

DogRotated dogmore

Data augmentation stems from our prior knowledge — for instance, we know in advance that ordinary photographs are invariant under horizontal flipping, so we use horizontal flipping to augment the data, thereby telling the model, through the augmented data, that it should be invariant to horizontal flips. Data augmentation is not universally applicable, though: ordinary photos can be flipped horizontally without changing their meaning, but this doesn't hold for images of text. This further illustrates that data augmentation is a scheme for incorporating our prior knowledge into the model.

SamplePairing

Now let's talk about SamplePairing. SamplePairing was proposed as a data augmentation technique, but it is extremely counterintuitive:

Given that sample $x_a$ has label $y_a$, randomly pick another sample $x_b$ from the training set, and let the label of $(x_a+x_b)/2$ still be $y_a$.

Yes, you read that correctly, and I didn't make a typo — it really is that simple, and it really is that counterintuitive. On paperweekly, a reader named Chen Taihong once commented:

This is the simplest paper I've ever seen in the CNN field.

Upon seeing this scheme, readers might have a series of questions popping up in their minds, and this series of questions is exactly where SamplePairing's counterintuitiveness lies. For instance, why is $(x_a+x_b)/2$ labeled as $y_a$ rather than $y_b$? Also, by our usual understanding of data augmentation, $(x_a+x_b)/2$ is no longer even a reasonable image — can such an "augmentation" still be useful?

CatDogAverage of cat and dog

First, regarding the asymmetry issue: notice that when the current sample is $x_a$ and we happen to randomly pick $x_b$, we let $(x_a+x_b)/2$ keep the label $y_a$; but when it's $x_b$'s turn, the random draw might also pick $x_a$, in which case the label of $(x_a+x_b)/2$ becomes $x_b$. So while it looks asymmetric, SamplePairing is actually symmetric, and it can be restated as follows:

Randomly pick two samples $x_a$ and $x_b$, with corresponding labels $y_a,y_b$, then randomly pick one of these labels — say the result is $y$ — and let the label of $(x_a+x_b)/2$ be $y$.

mixup

From this restated version of SamplePairing, it isn't hard to see that if training is sufficiently thorough, the model's output for $(x_a+x_b)/2$ should in theory be split evenly between $y_a,y_b$. That is, if $y_a,y_b$ denote the one-hot vectors of the respective categories, then the output for $(x_a+x_b)/2$ should be $(y_a+y_b)/2$.

Given this, why not mix things in a more randomized way? Suppose $U(\varepsilon)$ is some random distribution over $[0,1]$; then each time, we sample $\varepsilon\sim U(\varepsilon)$ randomly, and let the output corresponding to $\varepsilon x_a + (1-\varepsilon) x_b$ be $\varepsilon y_a + (1-\varepsilon) y_b$. In the original paper, $U(\varepsilon)$ is taken to be a $\beta$ distribution, but I feel this actually introduces more hyperparameters than necessary — it might as well just be a uniform distribution.

Compared to SamplePairing, mixup's approach is somewhat "gentler" and more convincing. SamplePairing always feels like it's making an abrupt "cut," whereas mixup is relatively more intuitive: since the input is a weighted combination via $\varepsilon: 1-\varepsilon$, it seems natural that the output should be combined the same way.

A Regularization Interpretation

Although mixup feels somewhat more reasonable, neither method answers a crucial question: after adding two images together, the result is no longer a sensible image at all — this is nothing like what we normally call data augmentation. So why does it still work?

Let's describe this more mathematically. Given a training pair $(x_1,y_1),(x_2,y_2),\dots,(x_n,y_n)$, we want to find a model $f$ such that $y=f(x)$. For tasks like image classification, given the strongly nonlinear nature of the problem, we generally use very deep networks to fit it. However, the deeper the network, the more prone it is to overfitting the training set.

Suppose the model already has the capacity to predict $y_a = f(x_a), y_b = f(x_b)$ correctly. mixup says this still isn't enough — the model must also, for $\varepsilon x_a + (1-\varepsilon) x_b$, output $\varepsilon y_a + (1-\varepsilon) y_b$, i.e.:

$$\varepsilon y_a + (1-\varepsilon) y_b = f\big(\varepsilon x_a + (1-\varepsilon) x_b\big)$$

Replacing $y_a,y_b$ with $f(x_a),f(x_b)$, we get:

$$\varepsilon f(x_a) + (1-\varepsilon) f(x_b) = f\big(\varepsilon x_a + (1-\varepsilon) x_b\big)$$

This is, in fact, a functional equation. If $\varepsilon,x_a,x_b$ are allowed to be arbitrary, then the solution to this functional equation is a "linear function" — that is, only a linear function can make the above equation hold identically. In other words, mixup requires the model $f$ to be a linear function.

We know that a linear function is equivalent to a single-layer neural network with no activation function — about as simple a model as you can get — whereas the model we actually use in practice is deep, has a huge number of parameters, and possesses strong nonlinear capacity. And the more parameters a model has, the more prone it is to overfitting. Given this, the meaning of mixup becomes clear:

mixup acts as a regularization term, pushing the model to be as close to a linear function as possible — that is, ensuring the model's predictions are as accurate as possible while keeping the model as simple as possible.

So mixup is in fact a very powerful model filter:

Among all models with similar performance, pick the one closest to being a linear function.

Closing Remarks

Now we can answer the original question: data augmentation techniques like SamplePairing and mixup only wear the appearance of data augmentation; in reality, they use the form of data augmentation to add a regularization term to the model, or equivalently, to prune the model.

So we no longer need to agonize over the question "how can data augmentation still be effective when the summed image is no longer even a sensible 'image'?" — because it isn't data augmentation at all.

Finally, let me close this post by irreverently rewriting a poem by Tang Bohu:

Others laugh at my haphazard augmentation; I laugh at those who fail to see through it.
No sensible image ever emerges — silently, invisibly, the model is being pruned.