[Reading List] Some Important Recent GAN Papers

【学习清单】最近比较重要的GAN进展论文 苏剑林 · 2018-12-26 · Machine Learning learning GAN generative model

This post simply lists what I consider to be the more important recent papers on GAN progress. It's basically the reading list I've been working through while studying GANs myself.

A Taste for Generative Models

GANs are a huge rabbit hole, and especially for an amateur player like me, you can dive in for a long time without much to show for it—especially now that big companies are throwing massive compute at ever-larger models, to the point where individuals almost can't play in this space anymore. But I always feel that only by actually getting your hands on generative models do you feel you've touched real machine learning. This holds true whether in images or in text. So I'm still willing to keep paying attention to generative models.

Of course, GAN is not the only option for generative modeling, but it's a very interesting one. In images, there are at least GAN, flow, and pixelrnn/pixelcnn as options, but in terms of potential, I still think GAN is the most promising—not just because of the results, but mainly because of the idea of adversarial training behind it. In text, the seq2seq mechanism is essentially already a probabilistic generative model, and models like pixelrnn are in fact designed in imitation of seq2seq. There is of course also research on using GANs for text generation (though it basically always involves reinforcement learning). In other words, generative models have produced quite a lot of results in NLP too—even if you mainly work on NLP, you will eventually run into generative models.

Alright, enough chit-chat—let's get the list out for everyone's reference, and as a memo for myself. more

Let the Results Speak

A Word Up Front

Not to put too fine a point on it, but right now in GAN research it's basically results that talk. No matter how perfect your theory is, if your experiments can't generate high-resolution images, it's hard to get accepted; no matter how ugly your approach looks, if your experimental results are good enough—if you can generate large, high-resolution images—everyone will flock to you.

One landmark event in GAN research was NVIDIA's Progressive Growing GANs from last year, which achieved 1024×1024 high-resolution face generation for the first time. Keep in mind that ordinary GANs already struggle to generate 128×128 faces, so achieving 1024-resolution generation counts as a real breakthrough. The papers listed below all managed to achieve 1024-resolution face generation in their own experiments. That result alone makes them worth our attention.

Of course, generating 1024-resolution images requires not just model improvements but also enormous compute, so it's hard for most people/labs to reproduce. The point of paying attention to these papers isn't that we need to reproduce image generation at this scale ourselves—it's that any model capable of generating images this large must have something worth learning from, and we might even come to understand where the bottlenecks of GANs lie, helping us avoid detours in our own research.

Paper List

"Progressive Growing of GANs for Improved Quality, Stability, and Variation"

Paper: https://papers.cool/arxiv/1710.10196

Reference implementation: https://github.com/tkarras/progressive_growing_of_gans

Brief introduction: This is the paper mentioned above that first achieved 1024-resolution face generation—Progressive Growing GANs, abbreviated PGGAN, from NVIDIA. As the name suggests, PGGAN achieves a smooth transition from low resolution to high resolution through a progressive architecture, allowing high-resolution models to be trained smoothly. The paper also offers its own insights and tricks regarding regularization and normalization, worth pondering. Of course, since it's progressive, it effectively requires training many models in sequence, so PGGAN is slow...

"Which Training Methods for GANs do actually Converge?"

Paper: https://papers.cool/arxiv/1801.04406

Reference implementation: https://github.com/LMescheder/GAN_stability

Brief introduction: This paper contains a lot of mathematical derivation around GAN training stability, ultimately arriving at a gradient penalty term simpler than WGAN-GP's. Anyone interested in GAN training stability should check it out. Besides the 1024 face generation, the paper also runs experiments on many other datasets, with quite good results—all trained directly end-to-end, no progressive architecture needed. My only puzzlement is: isn't this penalty term just a special case of the one in WGAN-div? Why doesn't the paper mention this?

"IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis"

Paper: https://papers.cool/arxiv/1807.06358

Reference implementation: (no open-source implementation with good results has been found yet)

Brief introduction: This is a VAE that can "introspect," improving upon VAE via adversarial training so that it can generate high-resolution images while simultaneously obtaining both an encoder and a generator. Besides being able to generate 1024-resolution images, what's even more worth noting is how ingenious this paper's design is. Models that can obtain both an encoder and a generator aren't unique in themselves—BiGAN can do that too—but what's distinctive about IntroVAE is that it directly uses the encoder as the discriminator, requiring no extra discriminator at all, i.e. saving a third of the parameter count outright. The deeper reasons behind this are worth analyzing and savoring carefully.

"Large Scale GAN Training for High Fidelity Natural Image Synthesis"

Paper: https://papers.cool/arxiv/1809.11096

Reference implementation: https://github.com/AaronLeong/BigGAN-pytorch

Brief introduction: This is the famous BigGAN. Although this paper doesn't provide 1024-resolution face generation results, it does provide generation results for natural scene images at 128, 256, and 512 resolution. Keep in mind that generating natural scene images is many times harder than generating CelebA faces—given that it can already generate 512-resolution natural scene images, we naturally have no doubt it could easily generate 1024-resolution faces too. There are already plenty of popular explainers about BigGAN online, so I won't repeat that here. The paper also proposes some of its own regularization tricks and shares a great deal of hyperparameter-tuning experience (which parameters, when adjusted, lead to good/bad changes), which is well worth referencing.

"Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow"

Paper: https://papers.cool/arxiv/1810.00821

Reference implementation: https://github.com/akanimax/Variational_Discriminator_Bottleneck

Brief introduction: This paper uses an information bottleneck to control the discriminator's fitting capacity, thereby acting as a regularizer and stabilizing GAN training. For an introduction to the information bottleneck, see my earlier post. Generally speaking, in ordinary supervised training, any technique that prevents overfitting can in theory be applied to the discriminator, and the information bottleneck is one such overfitting-prevention technique. Of course, as the title suggests, the paper isn't content to apply this only to GANs—besides the 1024 face generation experiment, it also runs experiments on imitation learning and reinforcement learning.

"A Style-Based Generator Architecture for Generative Adversarial Networks"

Paper: https://papers.cool/arxiv/1812.04948

Reference implementation: https://github.com/NVlabs/stylegan

Brief introduction: This is the new GAN generator architecture released just a few days ago, which many articles have dubbed "GAN 2.0"—again from NVIDIA, again from the authors of PGGAN, again following the PGGAN paradigm, only with a new generator architecture. They already achieved 1024-resolution generation a year ago, so of course this one is no exception. This new generator architecture is said to draw on style-transfer models, hence the name "Style-Based Generator." Having read it, I'd say it's basically the architecture of a conditional GAN (CGAN), except the condition and the noise have been swapped. In simple terms, the noise is treated as the condition, and the condition is treated as the noise, and then this is plugged into the CGAN framework. Judging from the result figures in the paper, this shift in thinking actually works quite well—I tried implementing it myself, and it does work, though with a bit of mode collapse. Let's all just wait for the official release. Incidentally, it was also the PGGAN authors who gave us the CelebA HQ dataset a year ago, and now they've given us a new dataset, FFHQ. Apparently both the dataset and the code will be open-sourced next January—let's wait and see.

Stabilize the Training First

A Word Up Front

Unlike supervised learning tasks—where, generally speaking, once you've designed a good model and have enough data and enough compute, you can get a good-enough model—GAN training is never just a matter of designing the model and being done with it. It's a matter of theory, model, and optimization all bound together. From a framework perspective, once GAN theory developed to WGAN, the theoretical framework was basically complete; everything after that has just been minor patching (including my own GAN-QP). From a model architecture perspective, DCGAN laid the groundwork, and the later ResNet + Upsampling approach became one of the standard frameworks; as for the just-released Style-Based Generator, no need to say more—so model architecture, too, is basically mature. So what's left?

It's optimization—that is, the training process. I feel that to truly master GANs, you need to carefully study their optimization process, perhaps analyzing their training trajectories from a dynamical-systems perspective. This might involve properties like existence, uniqueness, and stability of solutions to differential equations, and might also involve knowledge of stochastic optimization processes. In short, the optimization process needs to be brought into the analysis of GANs before GANs can truly become a complete theory.

The papers below analyze GAN training issues from different angles and offer their own solutions—worth a read.

Paper List

"Stabilizing Training of Generative Adversarial Networks through Regularization"

Paper: https://papers.cool/arxiv/1705.09367

Brief introduction: Derives a regularization term for GANs via a noise-injection approach; the derivation is, in theory, applicable to any f-GAN. Judging by the result figures in the paper, the results are pretty good.

"GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium"

Paper: https://papers.cool/arxiv/1706.08500

Brief introduction: Proposes the TTUR training strategy. The basic idea: whereas we usually alternate training the discriminator and generator a different number of times per iteration but with the same learning rate, we could instead train each of them once per iteration but with different learning rates—which is obviously more time-efficient. However, from a quick read, although the paper is theory-heavy, its theoretical foundation actually comes from another existing paper, "Stochastic Approximation with Two Time Scales," so in a sense this paper is just repeatedly applying that pre-existing theoretical result, which feels a bit monotonous.

"Which Training Methods for GANs do actually Converge?"

Paper: https://papers.cool/arxiv/1801.04406

Brief introduction: Already introduced above, but I'm listing it again here because it's simply too classic. This feels like required reading for anyone studying GAN training stability, understanding GAN training issues from the perspective of differential equations. In its stability analysis, this paper mainly draws on two other papers—one is its own "prequel" (by the same author), called The Numerics of GANs, and the other is Gradient Descent GAN Optimization Is Locally Stable—both classics.

"Spectral Normalization for Generative Adversarial Networks"

Paper: https://papers.cool/arxiv/1802.05957

Brief introduction: Implements a Lipschitz constraint on the discriminator via spectral normalization—arguably the most elegant method currently available for implementing this constraint. Spectral normalization is now also very widely used, so it's worth mentioning. See also my earlier post for a related introduction.

"Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect"

Paper: https://papers.cool/arxiv/1803.01541

Brief introduction: Adds a new regularization term to WGAN-GP. The idea behind this term is quite simple: it directly uses the Lipschitz constraint (in difference form) as a regularization term, similar to the extra quadratic term in GAN-QP's discriminator. Judging by the training curves in the paper, it's somewhat more stable than plain WGAN-GP.

More Additions Welcome

That's the whole list for this round—ten papers in total, conveniently enough. Given the limits of my own reading, there may well be omissions; if you have other recommendations, please feel free to raise them in the comments.

PS: Readers who only care about NLP needn't feel left out—some NLP posts will be coming out soon (^_^)