Cascading Rejection: A Simple Yet Effective Way to Improve GAN Performance

级联抑制：提升GAN表现的一种简单有效的方法 苏剑林 · 2019-12-01 · Machine Learning geometry GAN generative model

While browsing arXiv yesterday, I came across a paper from South Korea with a name that's refreshingly to the point: A Simple yet Effective Way for Improving the Performance of GANs]. Opening it up, I found the content equally concise: it proposes a way to strengthen the discriminator of a GAN, which brings a certain improvement to the GAN's generation metrics.

The authors call this method "Cascading Rejection." I'm not quite sure how best to translate it—dropping it into Baidu Translate gives "级联抑制" (cascading suppression/rejection), and thinking about it, that does seem to capture the flavor, so I'll just go with that for now. The reason I'm writing about this method isn't that it's especially powerful, but that I find its geometric meaning quite interesting, and it seems to carry some suggestive implications.

Orthogonal Decomposition

The discriminator of a GAN typically passes the input through several convolutional layers, then uses flatten or pooling to obtain a fixed-length vector $\boldsymbol{v}$, which is then combined via inner product with a weight vector $\boldsymbol{w}$ to produce a scalar score (setting aside details like bias terms and activation functions for now):

\begin{equation}D(\boldsymbol{x})=\langle \boldsymbol{v},\boldsymbol{w}\rangle\end{equation}

In other words, $\boldsymbol{v}$ serves as the representation of the input image, and the magnitude of the inner product between $\boldsymbol{v}$ and $\boldsymbol{w}$ is used to judge how "real" the image is. more

However, $\langle \boldsymbol{v},\boldsymbol{w}\rangle$ depends only on the projection component of $\boldsymbol{v}$ onto $\boldsymbol{w}$. In other words, if we fix $\langle \boldsymbol{v},\boldsymbol{w}\rangle$ and $\boldsymbol{w}$, then $\boldsymbol{v}$ can still vary a great deal, as shown in the left figure below.

Vectors v with the same inner product with w can still differ greatly The projection and perpendicular components of v

Suppose we take the view that when $\langle \boldsymbol{v},\boldsymbol{w}\rangle$ equals a certain value, the image is deemed real. The problem is that $\boldsymbol{v}$ can vary so widely—does every possible $\boldsymbol{v}$ really correspond to a genuine image? Clearly not necessarily. This exposes the shortcoming of scoring via an inner product: it only takes into account the projection component along $\boldsymbol{w}$, while ignoring the perpendicular component (as shown in the right figure above):

\begin{equation}\boldsymbol{v}-\Vert \boldsymbol{v}\Vert \cos(\boldsymbol{v},\boldsymbol{w}) \frac{\boldsymbol{w}}{\Vert \boldsymbol{w}\Vert}=\boldsymbol{v}- \frac{\langle\boldsymbol{v},\boldsymbol{w}\rangle}{\Vert \boldsymbol{w}\Vert^2}\boldsymbol{w}\end{equation}

Given this, a natural idea arises: can we use another parameter vector to classify this perpendicular component once more? Clearly we can—and moreover, this reclassification of the perpendicular component will itself produce a new perpendicular component, so the process can be iterated:

\begin{equation}\left\{\begin{aligned}&\boldsymbol{v}_1=\boldsymbol{v}\\ &D_1(\boldsymbol{x})=\langle \boldsymbol{v}_1,\boldsymbol{w}_1\rangle\\ &\boldsymbol{v}_2 = \boldsymbol{v}_1- \frac{\langle\boldsymbol{v}_1,\boldsymbol{w}_1\rangle}{\Vert \boldsymbol{w}_1\Vert^2}\boldsymbol{w}_1\\ &D_2(\boldsymbol{x})=\langle \boldsymbol{v}_2,\boldsymbol{w}_2\rangle\\ &\boldsymbol{v}_3 = \boldsymbol{v}_2- \frac{\langle\boldsymbol{v}_2,\boldsymbol{w}_2\rangle}{\Vert \boldsymbol{w}_2\Vert^2}\boldsymbol{w}_2\\ &D_3(\boldsymbol{x})=\langle \boldsymbol{v}_3,\boldsymbol{w}_3\rangle\\ &\boldsymbol{v}_4 = \boldsymbol{v}_3- \frac{\langle\boldsymbol{v}_3,\boldsymbol{w}_3\rangle}{\Vert \boldsymbol{w}_3\Vert^2}\boldsymbol{w}_3\\ &\qquad\vdots\\ &D_N(\boldsymbol{x})=\langle \boldsymbol{v}_N,\boldsymbol{w}_N\rangle\\ \end{aligned}\right.\end{equation}

Analysis and Reflection

At this point, I've essentially covered the core idea of the original paper; what remains are some operational details. First, having obtained $N$ scores $D_1(\boldsymbol{x}),D_2(\boldsymbol{x}),\dots,D_N(\boldsymbol{x})$, each score can have the discriminator loss applied to it (either directly using a hinge loss, or a cross-entropy after a sigmoid activation), and finally these $N$ losses are combined via a weighted average to form the final discriminator loss. This alone brings a performance improvement to the GAN. The authors further extend this to CGANs, also obtaining good results.

Experimental results of the GAN technique proposed in the paper

Compared to the experimental results, I think the deeper significance of this trick is more worth paying attention to. In fact, this idea could arguably be applied to general classification problems, not just GANs. Since the perpendicular components are iteratively folded into the prediction, we can think of the parameters $\boldsymbol{w}_1,\boldsymbol{w}_2,\dots,\boldsymbol{w}_N$ as representing $N$ distinct "viewpoints," with each classification corresponding to a judgment made from a different viewpoint.

This brought to mind Hinton's Capsule] networks. Although the form is quite different, there seems to be a shared underlying intuition: Capsules aim to represent an entity with a vector rather than a scalar, and here, "cascading rejection" likewise produces classification results from multiple angles through repeated orthogonal decomposition—that is, determining whether a vector belongs to a class requires giving multiple scores rather than just one, which also carries a whiff of "using a vector instead of a scalar."

Unfortunately, when I ran a simple experiment along these lines (on CIFAR-10), I found that the validation accuracy dropped slightly (note this doesn't contradict the GAN results—the improvement in GAN performance comes from making the discrimination task harder, whereas a supervised classification model doesn't want the task made harder). On the plus side, the degree of overfitting also decreased (i.e., the gap between training and validation accuracy narrowed). Of course, my experiment was too crude to draw any rigorous conclusions. Still, I think that given its clear geometric meaning, this trick is worth further thought.

Summary

This post introduced a technique with a clear geometric interpretation for improving GAN performance, and further discussed its potential broader value.