Designing GANs: Another GAN Production Workshop
In a 2018 post, A Brief Introduction to f-GAN: A Production Workshop for GAN Models, I introduced f-GAN and described it as a "production workshop" for GAN models—meaning that it can churn out many different forms of GANs following a fixed procedure. A few days ago I came across a new paper on arXiv, Designing GANs: A Likelihood Ratio Approach (referred to below simply as Designing GANs, or "the original paper"), and found that it does essentially the same thing as f-GAN, but takes a completely different route (though, as it turns out, the two roads lead to the same place). The paper is quite interesting, so let me share it here.
Recap of f-GAN
From A Brief Introduction to f-GAN: A Production Workshop for GAN Models, we know that the first step of f-GAN is to find a function $f$ satisfying the following conditions:
1. $f$ is a mapping from non-negative reals to reals ($\mathbb{R}^* \to \mathbb{R}$);
2. $f(1)=0$;
3. $f$ is a convex function.
Once such a function is found, one can construct a probability $f$ divergence, and then use a technique called "convex conjugation" to convert this $f$ divergence into another form (one involving $\max$, generally called the dual form). Then, by $\min$ this divergence, one obtains a min-max procedure, which gives birth to a GAN model. Incidentally, f-GAN represents a whole family of GAN models, but does not include WGAN—although the derivation of WGAN actually follows a similar set of steps, except that it uses the Wasserstein distance as its probability metric, and the method for converting the Wasserstein distance into its dual form is different. For details, see From Wasserstein Distance and Duality Theory to WGAN.
Logically, there's nothing wrong with f-GAN, but according to the procedure it prescribes, we always have to first find an $f$ divergence and only afterward convert it into the dual form. The natural question is: since we only need the dual form in the end, why not just work directly in the dual space from the start? I raised this question before, in the post A GAN Without Lipschitz Constraints That Doesn't Suffer From Vanishing Gradients—Care to Take a Look?, though at the time I only discussed the proof of the probability divergence in the dual space, without giving a method for constructing the probability divergence itself. Designing GANs is exactly what fills in this gap.
Designing GANs
In this section we'll explore the ideas and methods in Designing GANs. Rather than following the somewhat verbose, textbook-style derivation of the original paper, this post will derive the results of Designing GANs by working backward step by step, which I think is easier to follow. Interestingly, understanding the whole derivation really only requires very basic calculus.
Total Variation
Let's take a probability divergence called Total Variation as an example, to get a first taste of the key points of analyzing and deriving probability divergences directly in the dual space.
First, we have
\begin{equation}|p-q|=\max_{t\in[-1, 1]} (p - q)t=\max_{t\in[-1, 1]} pt - qt\label{eq:tv-base}\end{equation}
So for probability distributions $p(x),q(x)$, we also have
\begin{equation}|p(x)-q(x)|=\max_{t(x)\in[-1, 1]} p(x)t(x) - q(x)t(x)\end{equation}
Integrating both sides gives (let's not worry for now about whether we can swap the order of integration and $\max$)
\begin{equation}\begin{aligned}\int|p(x)-q(x)|dx=&\,\max_{t(x)\in[-1, 1]} \int \big[p(x)t(x) - q(x)t(x)\big]dx\\ =&\,\max_{t(x)\in[-1, 1]} \mathbb{E}_{x\sim p(x)}[t(x)] - \mathbb{E}_{x\sim q(x)}[t(x)] \end{aligned}\end{equation}
Here $\int|p(x)-q(x)|dx$ is called the Total Variation between the two probability distributions, so through this process we've derived the dual form of Total Variation. If we fix $p(x)$ and let $q(x)$ approach $p(x)$, we can minimize the Total Variation, i.e.
\begin{equation}\min_{q(x)}\int|p(x)-q(x)|dx = \min_{q(x)}\max_{t(x)\in[-1, 1]} \mathbb{E}_{x\sim p(x)}[t(x)] - \mathbb{E}_{x\sim q(x)}[t(x)]\label{eq:tv-gan}\end{equation}
This gives us a GAN formulation.
Looking back at the whole process, setting aside any prior familiarity with Total Variation, we can see that the core of the whole procedure is really equation $\eqref{eq:tv-base}$—once we have equation $\eqref{eq:tv-base}$, everything that follows is more or less automatic. So what's special about equation $\eqref{eq:tv-base}$? In fact, it can be generalized as follows:
Goal 1 Find a function $\phi(t),\psi(t)$ and some range $\Omega$ such that
\begin{equation}d(p, q) = \max_{t\in\Omega} p\phi(t)+q\psi(t)\end{equation}
and $d(p,q)\geq 0$ as well as $d(p,q)=0\Leftrightarrow p=q$.
Given such a $\phi(t),\psi(t)$, we can derive a GAN similar to $\eqref{eq:tv-gan}$:
\begin{equation}\min_{q(x)}\int d(p(x), q(x)) dx = \min_{q(x)}\max_{t(x)\in \Omega} \mathbb{E}_{x\sim p(x)}[\phi(t(x))] + \mathbb{E}_{x\sim q(x)}[\psi(t(x))]\label{eq:gan}\end{equation}
Finding the Maximum by Differentiation
Note that in "Goal 1," $p,q$ is just a non-negative real number, and $\phi(t),\psi(t)$ is just a scalar function, so the whole goal is purely an extremum problem for a function of one variable—already considerably simplified. In fact, if we set $r=q/p\in[0,+\infty)$, it can be reduced to the even more concise "Goal 2":
Goal 2 Find a function $\phi(t),\psi(t)$ and some range $\Omega$ such that
\begin{equation}d(r) = \max_{t\in\Omega} \phi(t)+r\psi(t)\end{equation}
and the minimum of $d(r)$ is attained at $r=1$.
Now let's examine Goal 2. For simplicity, let's assume $\phi(t),\psi(t)$ are both smooth functions, so that we can find the maximum of $\phi(t)+r\psi(t)$ by differentiation. This assumption is already fairly representative; when there are points that aren't smooth, we can approximate with smooth functions and then take a limit, e.g. $\text{sign}(x)=\lim\limits_{k\to+\infty}\tanh(kx)$.
Under this assumption, to find the maximum of $\phi(t)+r\psi(t)$, we first take the derivative and set it to zero:
\begin{equation}\phi'(t)+r\psi'(t)=0\quad\Rightarrow\quad r = -\frac{\phi'(t)}{\psi'(t)}\triangleq \omega^{-1}(t)\label{eq:max0}\end{equation}
Assume this equation has a unique solution, denoted $t=\omega(r)$, so in the end $-\frac{\phi'(t)}{\psi'(t)}$ is just $\omega^{-1}(t)$, the inverse function of $\omega(r)$ (to be clear, this is the inverse function, not the reciprocal). Also, since $r\in[0,+\infty)$, we have $t\in \Omega=\omega([0,\infty))$, meaning that the range $\Omega$ over which $t$ takes values is precisely the range of $\omega(r)$. Furthermore, since it can be inverted, $\omega(r)$ must be either strictly increasing or strictly decreasing; without loss of generality, assume $\omega(r)$ is strictly increasing, so $\omega^{-1}(t)$ is also strictly increasing.
A derivative of zero only tells us that $t$ is an extremum, not necessarily a maximum. Let's pin down the condition under which it is indeed the maximum. We now have
\begin{equation}\phi'(t)+r\psi'(t)=\big(r-\omega^{-1}(t)\big)\psi'(t)\end{equation}
Note that $r-\omega^{-1}(t)$ is strictly decreasing, so it can have only one zero, and it must be positive before that zero and negative after. To make sure the full derivative has this same property, we set $\psi'(t)\triangleq \rho(t) > 0$, i.e. require $\rho(t)$ to be always positive. Then the derivative of $\phi(t)+r\psi(t)$ is continuous, positive-then-negative, so $\phi(t)+r\psi(t)$ has only one extremum, and that extremum is a maximum.
Finding the Minimum by Differentiation
Let's first summarize what we have so far: we assume $\omega(r)$ is strictly increasing, and that $\rho(t)$ is always positive when $t\in \Omega$, satisfying the relation
\begin{equation}\left\{\begin{aligned}\phi'(t)=&\,-\omega^{-1}(t)\rho(t)\\ \psi'(t)=&\,\rho(t)\end{aligned} \right.\end{equation}
In that case, $\phi(t)+r\psi(t)$ has a unique maximizer $t=\omega(r)$, which is indeed the maximum. For Goal 2, this covers only half the requirement. We now need to check whether, as a function of $t=\omega(r)$, $d(r)=\phi(\omega(r))+r\psi(\omega(r))$ satisfies the remaining property we want (namely, that the minimum of $d(r)$ is attained at $r=1$).
Continuing to differentiate:
\begin{equation}d'(r)=\big[\phi'(\omega(r))+r\psi'(\omega(r))\big]\omega'(r)+\psi(\omega(r))=\psi(\omega(r))\label{eq:d-r}\end{equation}
where the second equality follows because, by equation $\eqref{eq:max0}$, the term in brackets is zero. Now only the term $\psi(\omega(r))$ remains, and recalling our assumption $\psi'(t)=\rho(t) > 0$, $\psi(t)$ is strictly increasing; and since we've also assumed $\omega(r)$ is strictly increasing, the composite function $\psi(\omega(r))$ is strictly increasing as well.
Let's now add one more assumption: $\psi(\omega(1))=0$, i.e. $d'(1)=0$, so that $r=1$ is an extremum of $d(r)$. Since $\psi(\omega(r))$ is continuous and strictly increasing, $d'(r)$ is negative-then-positive, so $r=1$ is a minimizer of $d(r)$, and indeed the minimum.
Delivering the GAN Model
At this point, our derivation is complete, and the condition we've obtained is:
Conclusion 1 If $\omega(r)$ is strictly increasing, $\Omega=\omega([0,+\infty))$, $\rho(t)$ is always positive when $t\in \Omega$, and satisfies the relation
\begin{equation}\left\{\begin{aligned}\phi'(t)=&\,-\omega^{-1}(t)\rho(t)\\ > \psi'(t)=&\,\rho(t)\end{aligned} > \right.\end{equation}
together with the condition $\psi(\omega(1))=0$, then any $\phi(t),\psi(t)$ meeting these conditions can be used to construct a GAN model of the form $\eqref{eq:gan}$.
Let's verify this with the original version of GAN:
\begin{equation}\begin{aligned}&\,\min_{q(x)}\max_{t(x)} \mathbb{E}_{x\sim p(x)}[\log (1-\sigma(t(x)))] + \mathbb{E}_{x\sim q(x)}[\log \sigma(t(x))]\\ =&\,\min_{q(x)}\max_{t(x)} \mathbb{E}_{x\sim p(x)}\left[-\log \left(1+e^{t(x)}\right)\right] + \mathbb{E}_{x\sim q(x)}\left[-\log \left(1+e^{-t(x)}\right)\right] \end{aligned}\label{eq:ori-gan}\end{equation}
where $\sigma(\cdot)$ is the sigmoid activation function. The above is the binary cross-entropy for a classification problem where real samples are labeled 0 and fake samples are labeled 1, and the right-hand side of the equation is the simplified result. From this we read off $\phi(t)=-\log \left(1+e^t\right)$, $\psi(t)=-\log \left(1+e^{-t}\right)$. Let's make a small adjustment and let $\psi(t)=\log 2-\log \left(1+e^{-t}\right)$—clearly this doesn't affect the original optimization problem. We now have $\rho(t)=\psi'(t)=\frac{1}{1+e^t}$, which is clearly always greater than 0, and $\omega^{-1}(t)=-\phi'(t)/\psi'(t)=e^t$, i.e. $t=\omega(r)=\log r$, which is also clearly strictly increasing. Finally, checking $\psi(\omega(1))$, we find it indeed equals 0.
From this example we can draw two corollaries:
1. The condition $\psi(\omega(1))=0$ isn't strictly necessary, because even if it isn't satisfied initially, we can always add a constant to $\psi(\omega(1))=0$... rather, to $\psi(t)$, to make it satisfy $\psi(\omega(1))=0$, and adding a constant doesn't change the original optimization problem;
2. If we swap the labels—letting real samples have label 1 and fake samples label 0—we get exactly the opposite property: the computed $\rho(t)$ is always negative, and $\omega(r)$ is strictly decreasing. This shows that Conclusion 1 as stated here is a sufficient, not necessary, condition for constructing a GAN.
Different forms of $\rho(t),\omega(r)$ can give the same GAN model. For instance, suppose we choose $t=\omega(r)=\frac{r}{r+1}$, giving $r=\omega^{-1}(t)=\frac{t}{1-t}$, and then choose $\rho(t)=\frac{1}{t}$. Then:
\begin{equation}\left\{\begin{aligned}\phi'(t)=&\,-\frac{t}{1-t}\times\frac{1}{t}\\ \psi'(t)=&\,\frac{1}{t}\end{aligned} \right.\end{equation}
which upon integration gives $\phi(t)=\log(1-t),\psi(t)$. Also, noting $\Omega=\omega([0,+\infty))=[0,1)$, and that $\rho(t)=\frac{1}{t}$ excludes $t=0$, the feasible domain is $(0,1)$ (in practice, boundary points can be ignored), so the resulting GAN is
\begin{equation}\min_{q(x)}\max_{t(x)\in (0,1)} \mathbb{E}_{x\sim p(x)}[\log (1-t(x))] + \mathbb{E}_{x\sim q(x)}[\log t(x)]\end{equation}
which is equivalent to the original GAN, just without an explicit activation function making $t(x)\in (0,1)$ hold.
Let's compute one more example. Choose $t=\omega(r)=\frac{1}{2}\log r$, i.e. $r=\omega^{-1}(t)=e^{2t}$, and let $\rho(t)=e^{-t}$. One can compute $\phi(t)=-e^t,\psi(t)=-e^{-t}$, giving a GAN variant:
\begin{equation}\min_{q(x)}\max_{t(x)} \mathbb{E}_{x\sim p(x)}\left[-e^{t(x)}\right] + \mathbb{E}_{x\sim q(x)}\left[-e^{-t(x)}\right]\end{equation}
The original paper uses the above conclusion to derive many more exotic GANs; interested readers can consult it directly. I won't repeat those derivations here.
Some Reflections and Extensions
Is there any connection between this method and f-GAN? Can this method be generalized further? Here are my own answers.
Connection with f-GAN
In the derivation above, the result of step $\max$ is $d(r)$, where $r=1$ is the minimizer of $d(r)$. Recalling Goal 1, and substituting $d(r)$ back into equation $\eqref{eq:gan}$, we find that our optimization objective is actually:
\begin{equation}\int p(x) d\left(\frac{q(x)}{p(x)}\right)dx\label{eq:f-gan}\end{equation}
Does this look familiar? Indeed, it looks exactly like the definition of an $f$ divergence. Moreover, recalling the derivation at $\eqref{eq:d-r}$, we know the derivative of $d(r)$ is strictly increasing, meaning $d(r)$ is convex, so the expression above really is an $f$ divergence! In other words, although this paper's authors took what looks like a completely different route, their results can all be derived through the f-GAN framework—nothing genuinely new is produced.
So does this mean this paper is entirely equivalent to f-GAN? Unfortunately, not quite. What the original paper achieves is only a subset of f-GAN: in other words, every GAN variant it can derive can also be derived by f-GAN, but not every GAN variant derivable by f-GAN can be derived by this paper's method.
Looking back at the whole derivation, its core idea is to directly generalize a "pointwise" metric formula to a "functional" one—for instance, generalizing the opening relation $|p-q|=0\Leftrightarrow p=q$ to $\int |p(x)-q(x)|dx=0\Leftrightarrow p(x)=q(x)$. Precisely because of this idea, the entire derivation can be carried out using only single-variable calculus. But the catch is: not every conclusion true for $\int d(p(x),q(x))dx=0\Leftrightarrow p(x)=q(x)$ implies the corresponding statement for $d(p,q)=0\Leftrightarrow p=q$. For example, the KL divergence is $\int p(x)\log \frac{p(x)}{q(x)}dx$, and it being 0 implies $p(x)=q(x)$, but $p\log\frac{p}{q}=0$ does not necessarily imply $p=q$. Hence, the original paper can't even derive the GAN corresponding to KL divergence.
Does this mean the original paper has no value? Looking purely at the "products," indeed it has none, since everything it produces, f-GAN can also produce. But we shouldn't only care about "products"—sometimes we should also care about the "production process." In fact, I think the academic value of the original paper lies in offering a reference method for analyzing and discovering GANs directly in the dual space, adding one more angle from which to understand GAN models.
It Can Actually Be Generalized Further
Whether we look at f-GAN or the original paper, the resulting generator and discriminator losses always take the same form, just with opposite sign/direction. But in practice, the GAN variants we actually use most often have generator and discriminator losses that are not the same. For instance, a more commonly used form of the original GAN is:
\begin{equation}\begin{aligned}&\,\max_{t(x)} \mathbb{E}_{x\sim p(x)}[\log (1-\sigma(t(x)))] + \mathbb{E}_{x\sim q(x)}[\log \sigma(t(x))]\\ &\,\min_{q(x)} \mathbb{E}_{x\sim q(x)}[-\log （1-\sigma(t(x))）] \end{aligned}\label{eq:ori-gan-2}\end{equation}
The same is true for LSGAN, Hinge GAN, and so on. So if we only consider variants where the generator and discriminator losses share the same form, we're missing something important.
In fact, this paper could go one step further and obtain even more results than f-GAN—unfortunately, it seems the authors talked themselves into a dead end without realizing it. Achieving this extra step is actually quite simple: in the process above, through step $\max\limits_{t\in\Omega} \phi(t)+r\psi(t)$ we solved for $t=\omega(r)$, and then substituted it back into the original $\phi(t)+r\psi(t)$ to get $d(r)$. But there's really no need to substitute back into the original expression—we could just as well substitute into any expression of the form $\alpha(t)+r\beta(t)$. According to equation $\eqref{eq:f-gan}$, combined with the requirements for an $f$ divergence listed at the start, it's enough that $d(r)=\alpha(\omega(r))+r\beta(\omega(r))$ be a convex function ($d(1)=0$ can always be achieved via a shift), or, per the earlier reasoning, that $d(r)$ be any function whose minimum is attained at $r=1$. Putting this all together:
Conclusion 2 If $\omega(r)$ is strictly increasing, $\Omega=\omega([0,+\infty))$, $\rho(t)$ is always positive when $t\in \Omega$, and satisfies the relation
\begin{equation}\left\{\begin{aligned}\phi'(t)=&\,-\omega^{-1}(t)\rho(t)\\ > \psi'(t)=&\,\rho(t)\end{aligned} > \right.\end{equation}
together with a function $\alpha(t),\beta(t)$ such that $d(r)=\alpha(\omega(r))+r\beta(\omega(r))$ is convex, or such that $d(r)$ is a function whose minimum is attained at $r=1$, then any $\phi(t),\psi(t),\alpha(t),\beta(t)$ meeting these conditions can be used to construct the following GAN model (where $\min\limits_{q(x)}$ has already dropped the $\alpha(t)$ term that is irrelevant to $q(x)$):
\begin{equation}\begin{aligned}\max_{t(x)\in \Omega} &\,\mathbb{E}_{x\sim p(x)}[\phi(t(x))] + \mathbb{E}_{x\sim q(x)}[\psi(t(x))]\\ > \min_{q(x)}&\,\mathbb{E}_{x\sim q(x)}[\beta(t(x))] > \end{aligned}\end{equation}
A Few Examples After Generalization
Using Conclusion 2 to construct GANs gives considerably more freedom—it allows us to build many examples that f-GAN simply cannot find, since it permits the generator and discriminator losses to differ.
For example, when computing the original GAN above, we obtained $t=\omega(r)=\log r$, and $r\log r$ happens to be a convex function, so we can let $\alpha(t)=0,\beta(t)=t$ (note that $\alpha(t)$ can be 0, but $\beta(t)$ cannot—think about why), giving $d(r)=r\log r$. The corresponding GAN is then
\begin{equation}\begin{aligned}\max_{t(x)} &\,\mathbb{E}_{x\sim p(x)}[\log (1-\sigma(t(x)))] + \mathbb{E}_{x\sim q(x)}[\log \sigma(t(x))]\\ \min_{q(x)}&\,\mathbb{E}_{x\sim q(x)}[t(x)] \end{aligned}\end{equation}
which is a rather useful GAN variant. Also, since $r\log r$ is convex, so is $(1+r)\log(1+r)$, so we can let $\alpha(t)=\beta(t)=\log(1+r)=\log\left(1+e^t\right)$, corresponding exactly to $d(r)=(r+1)\log(1+r)$, with $\log\left(1+e^t\right)=-\log(1-\sigma(t))$. In this case, the corresponding GAN is $\eqref{eq:ori-gan-2}$—this is actually the more commonly used original version of GAN, and it works better than $\eqref{eq:ori-gan}$.
Let's give another example. Take $t=\omega(r)=\frac{a + b r}{1 + r}$, where we set $b > a$ by convention, so $\Omega=(a,b)$ (again ignoring boundary points for now), and $r=\omega^{-1}(t)=\frac{t-a}{b-t}$. We take $\rho(t)=2(b-t)$, which satisfies the always-positive requirement. Solving this gives $\phi(t)=-(t-a)^2,\psi(t)=-(t-b)^2$. Next, take $d(r)=\frac{(r-1)^2}{r+1}$, which clearly attains its minimum at $r=1$, then set $\alpha(t)=\beta(t)$ and try to work backward to the form of $\beta(t)$, i.e. $d(r)=(1+r)\beta(t)$, from which we derive $\beta(t)=\left(\frac{r-1}{r+1}\right)^2$; substituting into $r=\frac{t-a}{b-t}$ gives $\beta(t)=\left(\frac{2}{b-a}t-\frac{a+b}{b-a}\right)^2$. For simplicity we can let $b-a=2$, so that $\beta(t)=\left(t-\frac{a+b}{2}\right)^2$. The resulting GAN form is
\begin{equation}\begin{aligned}\max_{t(x)\in (a,b)} &\,\mathbb{E}_{x\sim p(x)}\left[-(t-a)^2\right] + \mathbb{E}_{x\sim q(x)}\left[-(t-b)^2\right]\\ \min_{q(x)}&\,\mathbb{E}_{x\sim q(x)}\left[\left(t-\frac{a+b}{2}\right)^2\right] \end{aligned}\end{equation}
This is in fact exactly LSGAN. Readers might wonder: LSGAN doesn't have the constraint $t(x)\in (a,b)$—so where does that come from here? In fact this constraint can be dropped, because even without it, the corresponding optimal solution still ends up lying in $(a,b)$.
Clearly, these GAN variants obtained from the generalized Conclusion 2 are quite valuable, and they are variants that f-GAN cannot obtain. So if the original paper had included this generalization, it would have been quite elegant.
Summary
This post has shared a paper that designs GAN models directly in the dual space, analyzed its connection to f-GAN, and then generalized the original paper's results in a simple way, enabling it to design an even wider variety of GAN models.
Finally, readers might wonder: f-GAN already churned out so many GANs, and now this paper churns out yet more GANs—yet in practice we only ever use a handful of them, so what's the point of producing so many? That's a matter of perspective. Work of this kind is more important for its methodological value than for any immediate practical payoff. But what I'd really like to say is:
I never claimed it had any particular value—I just found it interesting, that's all.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.