When Generative Models Run Rampant: Should the Internet Fear "Mad Cow Disease"?

当生成模型肆虐：互联网将有“疯牛病”之忧？ 苏剑林 · 2023-07-14 · Machine Learning generative model

As we all know, whether in text or vision, all kinds of generative models are "running rampant" across the internet with unstoppable momentum. While everyone understands that true general artificial intelligence (AGI) still has a long way to go, this hasn't stopped people from increasingly frequently using generative models to create and share content. You've surely noticed that many online articles now come with illustrations generated by Stable Diffusion; you've surely noticed that many news pieces are increasingly showing shades of ChatGPT's style. This seemingly harmless trend is quietly raising a question: should we be wary of the generative-model-produced data flooding the internet?

A recent paper, Self-Consuming Generative Models Go MAD, reveals a worrying possibility: the unchecked expansion of generative models on the internet could trigger a digital-age version of "mad cow disease." In this post we'll go through this paper together and discuss its potential implications. more

"Eating Itself"

On one hand, as people use generative models more and more frequently, this will cause the amount of content on the internet created by generative models to keep growing. On the other hand, generative models themselves keep getting updated and iterated, and the data used to train them is also scraped from the internet — so it's easy to imagine that, going forward, an ever-larger fraction of each new training set will consist of content produced by earlier generative models. In other words, each subsequent generation of models may no longer have enough fresh data during training, and will instead be trained purely on data it produced itself — colloquially, "eating itself." This will cause the quality or diversity of the models to progressively deteriorate. The original paper calls this phenomenon "Model Autophagy Disorder" (MAD).

Coincidentally, a similar example once occurred in biology. Cows are herbivores; however, in order to boost nutritional supply, some livestock farmers ground up the remains of other cattle (including brain tissue) and mixed it into feed. At the time this seemed like a clever move, but it unexpectedly led to the emergence and widespread spread of "mad cow disease." This case illustrates that long-term "self-consumption" can allow harmful factors to accumulate within an organism, and once they reach a certain threshold, they may even trigger catastrophic disease.

For this reason, we likewise need to consider whether the rampant spread of generative models could trigger a similar "mad cow disease" on the internet — this could lead not only to the homogenization of information, making all kinds of content increasingly formulaic and lacking in originality and diversity, but could also give rise to a series of unforeseeable problems.

Declining Diversity

Some readers might wonder: isn't a generative model just a simulation of the true data distribution? Even if we continuously train on data produced by generative models across successive iterations, shouldn't this just be repeatedly reproducing the true data distribution — so how could it lead to a loss of diversity?

There are several reasons behind this. First, the data used to train generative models is often not drawn directly from the true distribution but has been processed by humans — for example, through denoising, normalization, and alignment. After such processing, the training set has already lost some of its diversity. For instance, the reason we can observe that many news reports or Zhihu answers carry a whiff of ChatGPT isn't due to the content itself, but rather the similarity of their format to ChatGPT's — which indicates that ChatGPT's training data and output style are both fairly distinctive and limited in range. Similarly, to reduce the training difficulty of image generation models, we often need to align images — for example, when training a face generation model, we commonly align all faces so that the eyes are positioned identically — and such operations also cause a loss of diversity.

Beyond this, another key factor is that, due to limitations of the generative model itself or of the training techniques used, no generative model can ever be perfect. As a result, we typically deliberately introduce techniques that sacrifice diversity in order to improve generation quality. For example, with GANs, Flow, and similar generative models, we often choose to reduce the variance of the sampling noise in order to obtain higher-quality generated results — this is the so-called truncation trick or annealing trick. Also, as discussed in A Ramble on Generative Diffusion Models (IX): Conditional Control of Generated Results, in diffusion models we typically introduce conditioning information to control the output — whether via Classifier-Guidance or Classifier-Free approaches, the introduction of extra conditions also restricts the diversity of generated results. In short, whenever a generative model falls short of perfection, in the process of balancing quality against diversity we are actively giving up some diversity.

Normal Distributions

To gain a deeper understanding of this phenomenon, let's next look at some concrete examples. To start, we'll consider the normal distribution, since it's simple enough that both derivation and analysis are much clearer. But as we'll see below, the results are already quite representative.

Suppose the true distribution is a multivariate normal distribution $\mathcal{N}(\boldsymbol{\mu}_0,\boldsymbol{\Sigma}_0)$, and the distribution we use for modeling is also normal, $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$. Then the process of training the model amounts to estimating the mean vector $\boldsymbol{\mu}$ and the covariance matrix $\boldsymbol{\Sigma}$ from the training set. We'll further assume that when training each generation of the generative model, only data produced by the previous generation is used — this is a fairly extreme assumption, but it's undeniable that as generative models become more widespread, this assumption gets closer and closer to being true.

Under these assumptions, we sample $n$ samples $\boldsymbol{x}_{t-1}^{(1)},\boldsymbol{x}_{t-1}^{(2)},\cdots,\boldsymbol{x}_{t-1}^{(n)}$ from the $t-1$-th generation model $\mathcal{N}(\boldsymbol{\mu}_{t-1},\boldsymbol{\Sigma}_{t-1})$, and use them to train the $t$-th generation model:

\begin{equation}\boldsymbol{\mu}_t = \frac{1}{n}\sum_{i=1}^n \boldsymbol{x}_{t-1}^{(i)},\quad \boldsymbol{\Sigma}_t=\frac{1}{n-1} \sum_{i=1}^n \big(\boldsymbol{x}_{t-1}^{(i)} - \boldsymbol{\mu}_t\big)\big(\boldsymbol{x}_{t-1}^{(i)} - \boldsymbol{\mu}_t\big)^{\top}\end{equation}

Note that if we add the truncation trick, then the $t$-th generation model becomes $\mathcal{N}(\boldsymbol{\mu}_t,\lambda\boldsymbol{\Sigma}_t)$, where $\lambda\in(0,1)$. It's then easy to see that the variance (diversity) of each generation will decay at a rate of $\lambda$, eventually reaching zero (complete loss of diversity). If we don't use the truncation trick (i.e., $\lambda=1$), does that mean everything is fine? Not really. By definition $\boldsymbol{\mu}_t = \frac{1}{n}\sum\limits_{i=1}^n \boldsymbol{x}_{t-1}^{(i)}$, since $\boldsymbol{x}_{t-1}^{(i)}$ are all obtained through random sampling, $\boldsymbol{\mu}_t$ is itself a random variable, and by the additivity property of normal distributions, it actually follows

\begin{equation}\boldsymbol{\mu}_t \sim \mathcal{N}\left(\boldsymbol{\mu}_{t-1},\frac{1}{n}\boldsymbol{\Sigma}_{t-1}\right)\quad\Rightarrow\quad\boldsymbol{\mu}_t \sim \mathcal{N}\left(\boldsymbol{\mu}_0,\frac{t}{n}\boldsymbol{\Sigma}_0\right)\end{equation}

It's easy to foresee that when $t$ is large enough, $\boldsymbol{\mu}_t$ itself will deviate noticeably from $\boldsymbol{\mu}_0$ — which corresponds to a collapse in quality, not merely a decline in diversity.

In short, introducing the truncation trick greatly accelerates the loss of diversity, and even without it, over long-term iterative training on finite samples, the generated distribution may still deviate significantly from the true original distribution. Note that the assumptions made in this normal distribution example are already much weaker than those for typical generative models — at the very least, its fitting capacity is guaranteed to be sufficient — and yet this still cannot avoid diversity decay or quality collapse. For real-world data and generative models with limited capacity, things should, in theory, only be worse.

Generative Models

For actual generative models, theoretical analysis is difficult, so we can only explore the results experimentally. The original paper conducted a very rich set of experiments, and the results basically align with the conclusions drawn from the normal distribution case: if the truncation trick is used, diversity is lost rapidly; even without the truncation trick, after repeated iterations the model will still inevitably drift away from the original distribution to some degree.

Here's an example with the truncation trick:

With truncation trick, generation 1 results

With truncation trick, generation 5 results

Here's an example without the truncation trick:

Without truncation trick, generation 1 results

Without truncation trick, generation 7 results

Of course, the assumption that "each round of iteration only uses data generated by the model from the previous round" is fairly extreme. The original paper also analyzed the case where each round includes a certain amount of real data, which itself breaks down into two sub-cases: (1) the sampled real data is fixed from the start and never changes; (2) fresh real data can be sampled at every iteration. The first approach is relatively easy to implement, but the paper shows that it can only slow down the rate of degradation, not fundamentally solve the problem; the second approach can indeed solve the degradation problem, but in practice it's very difficult to reliably distinguish real data from model-generated data.

Summary

This post has examined the consequences that may arise when various generative models "run rampant" across the internet at scale. When generative models are repeatedly updated and iterated using data they themselves generated, this may lead to severe homogenization of information and a loss of diversity — similar to the "mad cow disease" that once emerged from cows "eating cows."