Idempotent Generative Networks (IGN): A GAN That Tries to Merge Discriminator and Generator into One

幂等生成网络IGN：试图将判别和生成合二为一的GAN 苏剑林 · 2024-01-31 · Machine Learning GAN generative model adversarial

A while back, a generative model called "Idempotent Generative Network (IGN)" attracted a fair amount of attention. It billed itself as a new kind of generative model independent of the existing VAE, GAN, flow, and diffusion families, with the notable feature of single-step sampling. Perhaps people are so tired of the multi-step sampling generation process of today's mainstream diffusion models that any hint of a method claiming single-step sampling easily catches everyone's attention. On top of that, the word "idempotent" in the name IGN added a layer of mystique, further inflating expectations — and it certainly caught my interest too. But I was busy with other things at the time, so I never got around to carefully reading the model details.

Recently I had a bit of free time, remembered that IGN paper I hadn't read, and dug it back out. But after reading it, I was rather puzzled: how is this a new model at all? Isn't it just a variant of GAN? The difference from a conventional GAN is that it merges the generator and discriminator into one. So does this "merging into one" bring some special benefit, like more stable training? Personally, I don't think so. Below I'll share how I came to understand IGN from the GAN perspective, along with the questions it raised for me.

Generative Adversarial Networks

I studied GAN (Generative Adversarial Network) systematically some years ago (you can find related posts under the GAN tag), but haven't followed it closely in recent years. So let me first give a brief review of GAN here, which will also make it easier to compare GAN and IGN in later sections. more

GAN has two basic components: the Discriminator and the Generator — vividly thought of as the "appraiser" and the "forger." The discriminator is responsible for telling apart real samples from the fake samples produced by the generator, while the generator is responsible for mapping simple random noise into target samples, using the feedback signal from the discriminator to improve its own generation quality. Through this ongoing "offense-defense" struggle, the generator's output quality keeps improving until the discriminator can no longer distinguish real from fake at all, achieving an effect indistinguishable from reality.

Taking WGAN as an example, the training objective of the discriminator $D_{\theta}$ is to widen the score gap between real and fake samples:

\begin{equation}\max_{\theta} D_{\theta}(G_{\varphi}(z)) - D_{\theta}(x)\label{eq:d-loss}\end{equation}

Here $x$ is a real sample from the training set, $z$ is random noise, $G_{\varphi}$ is the generator, and $G_{\varphi}(z)$ is naturally the fake sample produced by the generator. The generator's training objective is to shrink the score gap between real and fake samples, i.e., to minimize the expression above. However, for the generator, the term $x$, which contains no generator parameters, is effectively a constant, so this simplifies to

\begin{equation}\min_{\varphi} D_{\theta}(G_{\varphi}(z))\label{eq:g-loss}\end{equation}

Beyond this there's also the matter of the Lipschitz constraint, but that's a detail we won't go into here; interested readers can further consult The Art of Mutual Sparring: From Scratch to WGAN-GP and From Wasserstein Distance and Duality Theory to WGAN.

Normally GAN training alternates between two losses, though sometimes it can also be written as a single loss being optimized in two opposite directions — some parameters undergo gradient descent, while others undergo gradient ascent. Such a training process, with opposing directions coexisting simultaneously (i.e., $\min\text{-}\max$), is typically unstable and prone to collapse. Or, even if training succeeds, there can be a mode collapse problem, manifesting as a lack of diversity — the generated results become overly uniform.

A Single Loss

Some readers might object: you just said GAN is trained by alternating between two losses, whereas IGN clearly uses a single loss — so how can you claim IGN is just a special case of GAN?

In fact, the way IGN writes its single loss is somewhat of a "sleight of hand." Following its style of writing, GAN can equally well be written as a single loss. How? It's simple: suppose $\theta',\varphi'$ is a copy of the weights of $\theta,\varphi$, i.e., $\theta'\equiv\theta,\varphi'\equiv\varphi$, but with gradients disabled on it. Then Equations $\eqref{eq:d-loss}$ and $\eqref{eq:g-loss}$ can be combined and written as:

\begin{equation}\min_{\theta,\varphi} D_{\theta}(x) - D_{\theta}(G_{\varphi'}(z)) + D_{\theta'}(G_{\varphi}(z))\label{eq:pure-one-loss}\end{equation}

The gradient of this combined expression with respect to $\theta,\varphi$ is identical to what you'd get from computing the two losses separately, so it's an equivalent implementation. So why call this a "sleight of hand"? Because there's no real trick here at all — it's purely renaming the original $\min\text{-}\max$ under a different symbol. If you actually implemented it this way, you'd need to repeatedly clone $D_{\theta'},G_{\varphi'}$ and detach its gradients, which severely hurts training efficiency.

In fact, to write GAN training as a genuinely practical single-loss formulation, you can refer to my earlier post Cleverly Cutting Gradients: Implementing GAN with a Single Loss, where by using the framework's built-in stop_gradient operator together with some gradient manipulation tricks, this goal can be achieved. Specifically, stop_gradient forcibly zeroes out the gradient of some part of the model — for example,

\begin{equation}\nabla_{\theta,\varphi} D_{\theta}(G_{\varphi}(z)) = \left(\frac{\partial D_{\theta}(G_{\varphi}(z))}{\partial\theta},\frac{\partial D_{\theta}(G_{\varphi}(z))}{\partial\varphi}\right)\label{eq:full-grad}\end{equation}

After adding the stop_gradient operator (abbreviated as $\color{skyblue}{\text{sg}}$), we get

\begin{equation}\nabla_{\theta,\varphi} D_{\theta}(\color{skyblue}{\text{sg}(}G_{\varphi}(z)\color{skyblue}{)}) = \left(\frac{\partial D_{\theta}(G_{\varphi}(z))}{\partial\theta},0\right)\label{eq:stop-grad}\end{equation}

So, using the stop_gradient operator, we can easily block the gradient flowing through the inner layer of a nested function (i.e., the gradient of $\varphi$). But what about the generator case, where we need to block the gradient of the outer layer of the nested function (i.e., the gradient of $\theta$)? There's no direct way to do this, but we can achieve it with a trick: subtracting Equation $\eqref{eq:stop-grad}$ from Equation $\eqref{eq:full-grad}$ gives

\begin{equation}\nabla_{\theta,\varphi} D_{\theta}(G_{\varphi}(z)) - \nabla_{\theta,\varphi} D_{\theta}(\color{skyblue}{\text{sg}(}G_{\varphi}(z)\color{skyblue}{)}) = \left(0,\frac{\partial D_{\theta}(G_{\varphi}(z))}{\partial\varphi}\right)\end{equation}

which achieves exactly this — blocking the gradient of the outer layer of the nested function. Combining the two expressions, we obtain one way of training GAN with a single loss:

\begin{equation}\begin{gathered} \min_{\theta,\varphi} \underbrace{D_{\theta}(x) - D_{\theta}(\color{skyblue}{\text{sg}(}G_{\varphi}(z)\color{skyblue}{)})}_{\text{removed}\varphi\text{gradient of}} + \underbrace{D_{\theta}(G_{\varphi}(z)) - D_{\theta}(\color{skyblue}{\text{sg}(}G_{\varphi}(z)\color{skyblue}{)})}_{\text{removed}\theta\text{gradient of}} \\[8pt] = \min_{\theta,\varphi} D_{\theta}(x) - 2 D_{\theta}(\color{skyblue}{\text{sg}(}G_{\varphi}(z)\color{skyblue}{)}) + D_{\theta}(G_{\varphi}(z))\end{gathered}\end{equation}

This way, there's no need to repeatedly clone the model, and gradient equivalence is achieved within a single loss.

Idempotent Generation

After all that setup, we can finally invite the protagonist of this post — the Idempotent Generative Network (IGN) — to take the stage. But before it formally appears, please bear with me a little longer as we first discuss IGN's motivation.

GAN has a notable characteristic: once training succeeds, usually only the generator is kept, while the discriminator is mostly discarded. However, in a well-balanced GAN, the generator and discriminator typically have parameter counts of the same order of magnitude, so discarding the discriminator means roughly half the total parameters go to waste — a bit of a shame. To address this, some works have tried adding an encoder into GAN and sharing part of the parameters between the discriminator and the encoder, improving parameter utilization. Among these, the simplest approach is my own O-GAN, which only slightly modifies the discriminator's structure and adds one extra loss term to turn the discriminator into an encoder, without adding any extra parameters or computation. It's one of the pieces of work I'm fairly satisfied with.

The title of this post gets straight to the point: IGN is a GAN that tries to merge the discriminator and generator into one, with the generator "playing both athlete and referee." From this angle, IGN can also be seen as another way to improve parameter utilization. First, IGN assumes that $z$ and $x$ are the same size, so the generator $G_{\varphi}$ has input and output of equal size — unlike a typical GAN, where the dimensionality of $z$ is usually much smaller than that of $x$. With this design of equal input and output sizes, the image itself can also be fed as input into the generator for further processing. IGN thus designs the discriminator as a reconstruction loss:

\begin{equation}\delta_{\varphi}(x) = \Vert G_{\varphi}(x) - x\Vert^2\label{eq:ign-d}\end{equation}

$\delta_{\varphi}$ reuses the original notation from the IGN paper and has no special meaning. This design fully reuses the generator's parameters without introducing any additional ones, which does look like an elegant design at first glance. Now, substituting this discriminator into Equation $\eqref{eq:pure-one-loss}$, we get

\begin{equation}\min_{\varphi}\underbrace{\delta_{\varphi}(x) - \delta_{\varphi}(G_{\varphi'}(z))}_{\text{discriminator loss}} + \underbrace{\delta_{\varphi'}(G_{\varphi}(z))}_{\text{generator loss}}\end{equation}

Doesn't this look exactly like the Final optimization objective in the original IGN paper? Of course, the original paper has two additional tunable coefficients, but in fact every term in Equation $\eqref{eq:pure-one-loss}$ can also be given its own tunable coefficient — this isn't anything special. So it's clear that IGN can be fully derived from GAN; it's simply a special case of GAN — even though the author claims not to have thought of IGN from the GAN perspective.

The term "idempotent" comes from the author's claim that, once IGN is successfully trained, the discriminator's score on real samples is 0, at which point $G_{\varphi}(x) = x$, and one can further derive

\begin{equation}G_{\varphi}(\cdots G_{\varphi}(x)) = \cdots = G_{\varphi}(G_{\varphi}(x)) = G_{\varphi}(x) = x\end{equation}

That is, applying $G_{\varphi}$ repeatedly to a real sample $x$ leaves the result unchanged — precisely the mathematical meaning of "idempotent." However, theoretically speaking, there's no way to guarantee that a GAN discriminator's loss on real samples is exactly zero, so true idempotence is hard to achieve in practice, and the original paper's experimental results bear this out.

My Own Analysis

A question well worth pondering is: why can the reconstruction loss $\eqref{eq:ign-d}$ successfully serve as a discriminator? Or put differently, there are countless expressions one could construct from $G_{\varphi}(x)$ and $x$ — can any of them serve as a discriminator?

Purely from the standpoint of "reconstruction loss as discriminator," IGN resembles EBGAN. But that doesn't mean EBGAN's success can explain IGN's success, because EBGAN's generator is independent of its discriminator, with no constraint that they fully share parameters — so EBGAN's success is "to be expected" and consistent with the original design philosophy of GAN. IGN is different, however, because its discriminator and generator completely share parameters, and GAN training is already inherently quite unstable, so it's very easy for "one to fail and both to fail together."

In my view, the reason IGN has a chance of not collapsing during training is that it happens to satisfy a kind of "self-consistency." First, the fundamental goal of GAN is that, for input noise $z$, $G_{\varphi}(z)$ should output a realistic image; as for IGN's design of "reconstruction loss as discriminator," even if the discriminator's optimal loss isn't exactly zero, it may still be close enough — that is, $G_{\varphi}(x)\approx x$ is approximately satisfied — and this simultaneously satisfies the condition that "for input image $x$, $G_{\varphi}(x)$ should also output a realistic image." In other words, regardless of the input, the output space is always that of realistic samples — this self-consistency is very important, since otherwise the generator might "fall apart" from having to generate in two conflicting directions at once.

Given all this, what real improvement does IGN offer over an ordinary GAN? Forgive my dullness, but I honestly can't see where the benefit lies. Take parameter utilization, for instance: at first glance, IGN's parameter sharing does seem to improve parameter efficiency, but in fact, to ensure that the generator $G_{\varphi}$ has equal-sized input and output, IGN uses an autoencoder structure — whose parameter count and computational cost are equal to the sum of the discriminator and generator of an ordinary GAN! In other words, far from reducing the parameter count, IGN actually increases the total computation by enlarging the generator.

I also ran a few simple experiments with IGN myself, and found that IGN training suffers from the same instability issues — arguably even worse, because the hard constraint of "parameter sharing + Euclidean distance" makes it easier to amplify this instability, leading to "one fails, all fail" rather than "one thrives, all thrive." Furthermore, because IGN's generator has equal-sized input and output, it loses the advantage that an ordinary GAN's generator has of projecting from a low-dimensional manifold into high-dimensional data space. IGN is likewise prone to mode collapse, and due to its reliance on Euclidean distance, its generated images tend to be blurrier, more like those from a VAE.

Summary

This post introduced the Idempotent Generative Network (IGN), which drew a fair amount of attention a while back, from the perspective of GAN, comparing its connections to and differences from GAN, and sharing my own analysis of it.