Talking About Generative Diffusion Models (IX): Conditional Control of Generation Results
The previous few posts have focused on fairly theoretical results; in this post, let's discuss a topic that is more practically valuable — conditional control of generation.
As generative models, diffusion models have followed a development history quite similar to that of VAEs, GANs, and flow-based models: unconditional generation appeared first, and conditional generation followed close behind. Unconditional generation is often used to explore the upper bound of model capability, whereas conditional generation is more about application-level utility, since it lets us control the output according to our own intentions. From DDPM up to now, a great deal of work on conditional diffusion models has appeared — one could even say that it was really conditional diffusion models that made diffusion models famous, as exemplified by the widely acclaimed text-to-image models DALL·E 2 and Imagen.
In this post, we'll do a brief study and summary of the theoretical foundations of conditional diffusion models.
Technical Analysis
Methodologically, approaches to conditional control of generation fall into two categories: post-hoc modification (Classifier-Guidance) and up-front training (Classifier-Free). more
For most people, training a SOTA-level diffusion model from scratch is far too costly, while training a classifier is still within reach. So the natural idea is to reuse someone else's pretrained unconditional diffusion model, and use a classifier to steer the generation process to achieve conditional control — this is the post-hoc Classifier-Guidance approach. For "deep-pocketed" companies like Google and OpenAI, on the other hand, which are not short of data or compute, it's more natural to inject the conditioning signal directly into the training process of the diffusion model itself, achieving better generation quality — this is the up-front Classifier-Free approach.
The Classifier-Guidance approach originally comes from Diffusion Models Beat GANs on Image Synthesis, where it was first used for class-conditional generation; later, More Control for Free! Image Synthesis with Semantic Diffusion Guidance generalized the notion of "classifier" so that generation could also be conditioned on images or text. The Classifier-Guidance approach has relatively low training cost (readers familiar with NLP may be reminded of the similarly-flavored PPLM model), but its inference cost is higher, and the fine-grained control it offers is usually not quite as precise.
As for the Classifier-Free approach, it originally comes from Classifier-Free Diffusion Guidance, and later eye-catching models such as DALL·E 2 and Imagen are basically built on top of it. It's worth noting that this paper was only posted on Arxiv last month, but it had actually already been accepted at NeurIPS 2021. Strictly speaking, the Classifier-Free approach involves no particular theoretical trick — it is simply the most straightforward way of building a conditional diffusion model. It appeared later mainly because retraining a diffusion model is expensive; but given ample data and compute, the Classifier-Free approach shows remarkably fine control over generation details.
Conditional Input
Put plainly, the Classifier-Free approach is expensive to train and has "no particular technical sophistication" of its own, so the bulk of what follows will focus on the Classifier-Guidance approach, with the Classifier-Free approach briefly introduced at the end.
Having gone through the analysis in the previous series of posts, readers will likely already know that the key step in building a generative diffusion model is constructing the generation process $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)$. For generation conditioned on an input $\boldsymbol{y}$, all we need to do is replace $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)$ with $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})$ — that is, add the input $\boldsymbol{y}$ into the generation process. In order to reuse an already-trained unconditional generative model $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)$, we apply Bayes' theorem to get
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{y}) = \frac{p(\boldsymbol{x}_{t-1})p(\boldsymbol{y}|\boldsymbol{x}_{t-1})}{p(\boldsymbol{y})}\end{equation}
Adding the condition $\boldsymbol{x}_t$ to every term gives
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y}) = \frac{p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)p(\boldsymbol{y}|\boldsymbol{x}_{t-1}, \boldsymbol{x}_t)}{p(\boldsymbol{y}|\boldsymbol{x}_t)}\label{eq:bayes-1}\end{equation}
Note that in the forward process, $\boldsymbol{x}_t$ is obtained from $\boldsymbol{x}_{t-1}$ by adding noise, and noise doesn't help classification at all, so including $\boldsymbol{x}_t$ brings no benefit to the classification term. Hence $p(\boldsymbol{y}|\boldsymbol{x}_{t-1}, \boldsymbol{x}_t)=p(\boldsymbol{y}|\boldsymbol{x}_{t-1})$, and therefore
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y}) = \frac{p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)p(\boldsymbol{y}|\boldsymbol{x}_{t-1})}{p(\boldsymbol{y}|\boldsymbol{x}_t)} = p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t) e^{\log p(\boldsymbol{y}|\boldsymbol{x}_{t-1}) - \log p(\boldsymbol{y}|\boldsymbol{x}_t)}\label{eq:bayes-2}\end{equation}
Approximating the Distribution
Readers who have already gone through Talking About Generative Diffusion Models (V): The General SDE Framework will probably find what follows familiar. But even without having read it, that's fine — we'll still go through the full derivation below.
When $T$ is sufficiently large, the variance of $p(\boldsymbol{x}_t|\boldsymbol{x}_{t-1})$ is small enough that the probability is noticeably greater than 0 only when $\boldsymbol{x}_t$ is close to $\boldsymbol{x}_{t-1}$. The converse also holds: it is only when $\boldsymbol{x}_t$ is close to $\boldsymbol{x}_{t-1}$ that $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})$ or $p(\boldsymbol{x}_t|\boldsymbol{x}_{t-1}, \boldsymbol{y})$ is noticeably greater than 0, so we only need to focus on the probability changes within this range. To that end, we use a Taylor expansion:
\begin{equation}\log p(\boldsymbol{y}|\boldsymbol{x}_{t-1}) - \log p(\boldsymbol{y}|\boldsymbol{x}_t)\approx (\boldsymbol{x}_{t-1} - \boldsymbol{x}_t)\cdot\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)\end{equation}
Strictly speaking there is also a term involving variation of $t$, but that term is unrelated to $\boldsymbol{x}_{t-1}$ and is just a constant that doesn't affect the probability over $\boldsymbol{x}_{t-1}$, so we won't write it out. Assuming we originally had $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)=\mathcal{N}(\boldsymbol{x}_{t-1};\boldsymbol{\mu}(\boldsymbol{x}_t),\sigma_t^2\boldsymbol{I})\propto e^{-\Vert \boldsymbol{x}_{t-1} - \boldsymbol{\mu}(\boldsymbol{x}_t)\Vert^2/2\sigma_t^2}$, then approximately we now have
\begin{equation}\begin{aligned} p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y}) \propto&\, e^{-\Vert \boldsymbol{x}_{t-1} - \boldsymbol{\mu}(\boldsymbol{x}_t)\Vert^2/2\sigma_t^2 + (\boldsymbol{x}_{t-1} - \boldsymbol{x}_t)\cdot\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)} \\ \propto&\, e^{-\Vert \boldsymbol{x}_{t-1} - \boldsymbol{\mu}(\boldsymbol{x}_t) - \sigma_t^2 \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t))\Vert^2/2\sigma_t^2} \end{aligned}\end{equation}
From this result we can see that $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})$ is approximately $\mathcal{N}(\boldsymbol{x}_{t-1};\boldsymbol{\mu}(\boldsymbol{x}_t) + \sigma_t^2 \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t),\sigma_t^2\boldsymbol{I})$, so all we need to do is change the sampling of the generation process to
\begin{equation}\boldsymbol{x}_{t-1} = \boldsymbol{\mu}(\boldsymbol{x}_t) \color{skyblue}{+} {\color{skyblue}{\underbrace{\sigma_t^2 \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)}_{\text{new term}}}} + \sigma_t\boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\end{equation}
This is the core result of the Classifier-Guidance approach. Note that the input to $p(\boldsymbol{y}|\boldsymbol{x}_t)$ is the noised sample $\boldsymbol{x}_t$, meaning we need a model capable of making predictions on noised samples. If we only have a model $p_o(\boldsymbol{y}|\boldsymbol{x})$ trained to predict on clean samples, then a reasonable choice for $p(\boldsymbol{y}|\boldsymbol{x}_t)$ is
\begin{equation}p(\boldsymbol{y}|\boldsymbol{x}_t) = p_{o}(\boldsymbol{y}|\boldsymbol{\mu}(\boldsymbol{x}_t))\end{equation}
That is, we use $\boldsymbol{\mu}(\cdot)$ to denoise $\boldsymbol{x}_t$ before feeding it into $p(\boldsymbol{y}|\boldsymbol{x}_t)$, which avoids the extra cost of training a classifier on noised samples.
Gradient Scaling
The original paper (Diffusion Models Beat GANs on Image Synthesis) found that introducing a scaling parameter $\gamma$ into the classifier gradient can better tune the generation results:
\begin{equation}\boldsymbol{x}_{t-1} = \boldsymbol{\mu}(\boldsymbol{x}_t) \color{skyblue}{+} \color{skyblue}{\sigma_t^2 \color{red}{\gamma}\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)} + \sigma_t\boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\label{eq:gamma-sample}\end{equation}
When $\gamma > 1$, the generation process makes greater use of the classifier signal, which increases the correlation between the generated result and the input signal $\boldsymbol{y}$, but correspondingly reduces diversity in the generated results; conversely, it decreases the correlation between the generated results and the input signal, but increases diversity.
How should we understand this parameter theoretically? The original paper proposed interpreting it as sharpening the distribution's concentration via a power operation, i.e., defining
\begin{equation}\tilde{p}(\boldsymbol{y}|\boldsymbol{x}_t) = \frac{p^{\gamma}(\boldsymbol{y}|\boldsymbol{x}_t)}{Z(\boldsymbol{x}_t)},\quad Z(\boldsymbol{x}_t)=\sum_{\boldsymbol{y}} p^{\gamma}(\boldsymbol{y}|\boldsymbol{x}_t)\end{equation}
As $\gamma$ increases, the prediction of $\tilde{p}(\boldsymbol{y}|\boldsymbol{x}_t)$ becomes closer and closer to a one-hot distribution; using it in place of $p(\boldsymbol{y}|\boldsymbol{x}_t)$ as the classifier for Classifier-Guidance, the generation process will tend to pick out samples with very high classification confidence.
However, while this perspective offers some reference value, it isn't entirely correct, because
\begin{equation}\nabla_{\boldsymbol{x}_t}\log \tilde{p}(\boldsymbol{y}|\boldsymbol{x}_t) = \gamma\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t) - \nabla_{\boldsymbol{x}_t} \log Z(\boldsymbol{x}_t) \neq \gamma\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)\end{equation}
The original paper mistakenly assumed that $Z(\boldsymbol{x}_t)$ is a constant, and so concluded that $\nabla_{\boldsymbol{x}_t} \log Z(\boldsymbol{x}_t)=0$; but in fact, when $\gamma\neq 1$, $Z(\boldsymbol{x}_t)$ explicitly depends on $\boldsymbol{x}_t$. The author has also given some thought to whether there's any way to fix this, but unfortunately found no results — it seems one can only rather reluctantly assume that the gradient properties at $\gamma=1$ (where $Z(\boldsymbol{x}_t)=1$) approximately generalize to the case of $\gamma\neq 1$.
Similarity-Based Control
In fact, the best way to understand $\gamma\neq 1$ is to abandon trying to interpret it through the Bayes' theorem expressions in equations $\eqref{eq:bayes-1}$ and $\eqref{eq:bayes-2}$, i.e., abandon deriving it from $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})$, and instead define it directly as
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y}) = \frac{p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t) e^{\gamma\cdot\text{sim}(\boldsymbol{x}_{t-1}, \boldsymbol{y})}}{Z(\boldsymbol{x}_t, \boldsymbol{y})},\quad Z(\boldsymbol{x}_t,\boldsymbol{y})=\sum_{\boldsymbol{x}_{t-1}} p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t) e^{\gamma\cdot\text{sim}(\boldsymbol{x}_{t-1}, \boldsymbol{y})}\end{equation}
where $\text{sim}(\boldsymbol{x}_{t-1}, \boldsymbol{y})$ is some similarity or correlation measure between the generated result $\boldsymbol{x}_{t-1}$ and the condition $\boldsymbol{y}$. From this perspective, $\gamma$ is directly built into the definition of $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})$, directly controlling the correlation between the result and the condition: the larger $\gamma$ is, the more the model tends to generate $\boldsymbol{x}_{t-1}$ that is more strongly correlated with $\boldsymbol{y}$.
To get an approximate result that can actually be sampled, we can expand around $\boldsymbol{x}_{t-1}=\boldsymbol{x}_t$ (we could also expand around $\boldsymbol{x}_{t-1}=\boldsymbol{\mu}(\boldsymbol{x}_t)$, in a manner similar to before):
\begin{equation}e^{\gamma\cdot\text{sim}(\boldsymbol{x}_{t-1}, \boldsymbol{y})}\approx e^{\gamma\cdot\text{sim}(\boldsymbol{x}_t, \boldsymbol{y}) + \gamma\cdot(\boldsymbol{x}_{t-1}-\boldsymbol{x}_t)\cdot\nabla_{\boldsymbol{x}_t}\text{sim}(\boldsymbol{x}_t, \boldsymbol{y})} \end{equation}
Assuming this approximation is sufficiently accurate, dropping the terms unrelated to $\boldsymbol{x}_{t-1}$, we get
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})\propto p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)e^{\gamma\cdot(\boldsymbol{x}_{t-1}-\boldsymbol{x}_t)\cdot\nabla_{\boldsymbol{x}_t}\text{sim}(\boldsymbol{x}_t, \boldsymbol{y})} \end{equation}
As before, substituting in $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)=\mathcal{N}(\boldsymbol{x}_{t-1};\boldsymbol{\mu}(\boldsymbol{x}_t),\sigma_t^2\boldsymbol{I})$ and completing the square, we obtain
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y})\approx \mathcal{N}(\boldsymbol{x}_{t-1}; \boldsymbol{\mu}(\boldsymbol{x}_t) + \sigma_t^2\gamma \nabla_{\boldsymbol{x}_t} \text{sim}(\boldsymbol{x}_t, \boldsymbol{y}),\sigma_t^2\boldsymbol{I}) \end{equation}
In this way we don't need to worry about the probabilistic meaning of $p(\boldsymbol{y}|\boldsymbol{x}_t)$ at all — we simply need to directly define the measure function $\text{sim}(\boldsymbol{x}_t, \boldsymbol{y})$. Here $\boldsymbol{y}$ is no longer restricted to being a "class label" either; it can be text, an image, or any other kind of input signal. The usual approach is to encode it into a feature vector using its own encoder, and then use cosine similarity:
\begin{equation}\text{sim}(\boldsymbol{x}_t, \boldsymbol{y}) = \frac{E_1(\boldsymbol{x}_t)\cdot E_2(\boldsymbol{y})}{\Vert E_1(\boldsymbol{x}_t)\Vert \Vert E_2(\boldsymbol{y})\Vert}\end{equation}
It should be pointed out that the intermediate $\boldsymbol{x}_t$ carries Gaussian noise, so the encoder $E_1$ generally cannot be a clean-data encoder taken directly off the shelf — it's better to fine-tune it on noised data first. Additionally, for style transfer tasks, one usually uses Gram matrix distance instead of cosine similarity; these choices are all up to the specific scenario. The above is a summary of the results from the paper More Control for Free! Image Synthesis with Semantic Diffusion Guidance; readers can refer to the original paper for further details.
The Continuous-Time Case
From the derivation above, we obtained that the correction term to the mean is either $\sigma_t^2 \gamma \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)$ or $\sigma_t^2\gamma \nabla_{\boldsymbol{x}_t} \text{sim}(\boldsymbol{x}_t, \boldsymbol{y})$, and both share a common feature: when $\sigma_t=0$, the correction term is also 0, so the correction fails to have any effect.
So can the $\sigma_t$ of the generation process actually equal 0? Certainly it can — for instance, the DDIM introduced in Talking About Generative Diffusion Models (IV): DDIM = A High-Level View of DDPM is precisely a generation process with zero variance. How should we perform conditional control generation in that case? Here we need the general SDE-based results introduced in Talking About Generative Diffusion Models (VI): The General ODE Framework. There, we introduced that for the forward SDE:
\begin{equation}d\boldsymbol{x} = \boldsymbol{f}_t(\boldsymbol{x}) dt + g_t d\boldsymbol{w}\end{equation}
the corresponding most general reverse SDE is
\begin{equation}d\boldsymbol{x} = \left(\boldsymbol{f}_t(\boldsymbol{x}) - \frac{1}{2}(g_t^2 + \sigma_t^2)\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x})\right) dt + \sigma_t d\boldsymbol{w}\end{equation}
Here we are free to choose the reverse-time variance $\sigma_t^2$; DDPM and DDIM can both be regarded as special cases of it, with $\sigma_t=0$ giving the generalized DDIM. We can see that the only input-dependent term in the reverse SDE is $\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x})$. To perform conditional generation, naturally we just need to replace it with $\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x}|\boldsymbol{y})$, and then, using Bayes' theorem, we have
\begin{equation}\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x}|\boldsymbol{y}) = \nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x}) + \nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{y}|\boldsymbol{x})\end{equation}
Under the usual parameterization we have $\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x}) = -\frac{\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)}{\bar{\beta}_t}$, so
\begin{equation}\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x}|\boldsymbol{y}) = -\frac{\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)}{\bar{\beta}_t} + \nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{y}|\boldsymbol{x}) = -\frac{\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \bar{\beta}_t\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{y}|\boldsymbol{x})}{\bar{\beta}_t}\end{equation}
This means that regardless of what value the generation variance takes, we simply need to replace $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ with $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \bar{\beta}_t\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{y}|\boldsymbol{x})$ to achieve conditional control generation. Thus, from the unified SDE perspective, we can obtain the most general form of the Classifier-Guidance result very simply and directly.
Without a Classifier
Finally, let's briefly introduce the Classifier-Free approach. It's actually very simple: it directly defines
\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{y}) = \mathcal{N}(\boldsymbol{x}_{t-1}; \boldsymbol{\mu}(\boldsymbol{x}_t, \boldsymbol{y}),\sigma_t^2\boldsymbol{I}) \end{equation}
Following the results from the earlier posts on DDPM, $\boldsymbol{\mu}(\boldsymbol{x}_t, \boldsymbol{y})$ is generally parameterized as
\begin{equation}\boldsymbol{\mu}(\boldsymbol{x}_t, \boldsymbol{y}) = \frac{1}{\alpha_t}\left(\boldsymbol{x}_t - \frac{\beta_t^2}{\bar{\beta}_t}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, \boldsymbol{y}, t)\right)\end{equation}
and the training loss function is
\begin{equation}\mathbb{E}_{\boldsymbol{x}_0,\boldsymbol{y}\sim\tilde{p}(\boldsymbol{x}_0,\boldsymbol{y}), \boldsymbol{\varepsilon}\sim\mathcal{N}(\boldsymbol{0}, \boldsymbol{I})}\left[\left\Vert\boldsymbol{\varepsilon} - \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}, \boldsymbol{y}, t)\right\Vert^2\right]\end{equation}
Its advantage is that the extra input $\boldsymbol{y}$ is introduced right from training time, and in principle, the more input information there is, the easier training becomes; its disadvantage is exactly the same thing — because the extra input $\boldsymbol{y}$ is introduced during training, every new signal condition requires retraining the entire diffusion model from scratch.
Notably, the Classifier-Free approach also mimics the Classifier-Guidance approach by introducing a $\gamma$ scaling mechanism to balance correlation against diversity. Specifically, the mean in equation $\eqref{eq:gamma-sample}$ can be rewritten as:
\begin{equation}\boldsymbol{\mu}(\boldsymbol{x}_t) + \sigma_t^2 \gamma \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t) = \gamma\left[\boldsymbol{\mu}(\boldsymbol{x}_t) + \sigma_t^2 \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)\right] - (\gamma - 1) \boldsymbol{\mu}(\boldsymbol{x}_t)\end{equation}
The Classifier-Free approach amounts to directly fitting $\boldsymbol{\mu}(\boldsymbol{x}_t) + \sigma_t^2 \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{y}|\boldsymbol{x}_t)$ with a model, so analogous to the expression above, we can also introduce the $w=\gamma - 1$ parameter into the Classifier-Free approach, using
\begin{equation}\tilde{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, \boldsymbol{y}, t) = (1 + w)\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, \boldsymbol{y}, t) - w \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\end{equation}
in place of $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, \boldsymbol{y}, t)$ for generation. So how do we get the unconditional $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$? We can introduce a new special input $\boldsymbol{\phi}$, whose corresponding target images are the entire set of images, and add it into the model's training — this way we can treat it as $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)=\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, \boldsymbol{\phi}, t)$.
Summary
This post gave a brief introduction to the theoretical results underlying conditional diffusion models, covering two main approaches: post-hoc modification (Classifier-Guidance) and up-front training (Classifier-Free). The former does not require retraining the diffusion model and can achieve simple control at low cost; the latter requires retraining the diffusion model, which is more costly, but can achieve much finer-grained control.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.