Generative Diffusion Models Explained (Part 11): The Unified Diffusion Model (Applications)

生成扩散模型漫谈（十一）：统一扩散模型（应用篇） 苏剑林 · 2022-09-21 · Machine Learning unification generative model DDPM diffusion

In Generative Diffusion Models Explained (Part 10): The Unified Diffusion Model (Theory), I claimed to have built a unified model framework (Unified Diffusion Model, UDM) that allows for more general diffusion mechanisms and data types. So does the UDM framework actually achieve this goal? This post demonstrates its generality through several concrete examples.

Recap of the Framework

First, UDM constructs the forward process by choosing a noise distribution $q(\boldsymbol{\varepsilon})$ and a transformation $\boldsymbol{\mathcal{F}}$:

\begin{equation}\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon}),\quad \boldsymbol{\varepsilon}\sim q(\boldsymbol{\varepsilon})\end{equation}

Then, sampling for the reverse process $\boldsymbol{x}_{t-1}\sim p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t)$ is achieved through the following decomposition:

\begin{equation}\hat{\boldsymbol{x}}_0\sim p(\boldsymbol{x}_0|\boldsymbol{x}_t)\quad \& \quad \boldsymbol{x}_{t-1}\sim p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0=\hat{\boldsymbol{x}}_0)\end{equation}

Here $p(\boldsymbol{x}_0|\boldsymbol{x}_t)$ is the model that uses $\boldsymbol{x}_t$ to estimate the probability of $\boldsymbol{x}_0$; it is generally approximated with a simple distribution $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$, and the training objective is essentially $-\log q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ or a simple variant thereof. When $\boldsymbol{x}_0$ is continuous data, $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ is generally taken to be a conditional normal distribution; when $\boldsymbol{x}_0$ is discrete data, $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ can be chosen as either an autoregressive model or a non-autoregressive model. more

As for $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0)$, the most basic choice is

\begin{equation}p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0) = p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_0)\quad\Leftrightarrow\quad \boldsymbol{x}_{t-1}=\boldsymbol{\mathcal{F}}_{t-1}(\boldsymbol{x}_0,\boldsymbol{\varepsilon})\end{equation}

Starting from this baseline, different optimized results can be obtained under different conditions. When $\boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon})$ is invertible with respect to $\boldsymbol{\varepsilon}$, we can solve for $\boldsymbol{\varepsilon} = \boldsymbol{\mathcal{F}}_t^{-1}(\boldsymbol{x}_0,\boldsymbol{x}_t)$ and thereby obtain a better deterministic sampling scheme:

\begin{equation}\boldsymbol{x}_{t-1} = \boldsymbol{\mathcal{F}}_{t-1}(\boldsymbol{x}_0,\boldsymbol{\mathcal{F}}_t^{-1}(\boldsymbol{x}_0,\boldsymbol{x}_t))\end{equation}

Furthermore, if $q(\boldsymbol{\varepsilon})$ is a standard normal distribution, we can obtain

\begin{equation}\quad\boldsymbol{x}_{t-1} = \boldsymbol{\mathcal{F}}_{t-1}(\boldsymbol{x}_0,\sqrt{1 - \tilde{\sigma}_t^2}\boldsymbol{\mathcal{F}}_t^{-1}(\boldsymbol{x}_0,\boldsymbol{x}_t) + \tilde{\sigma}_t \boldsymbol{\varepsilon})\end{equation}

Hot Diffusion

In this section, we show that "hot diffusion" is a special case of UDM. Here "Hot Diffusion" refers to the mainstream diffusion models discussed earlier, such as DDPM and DDIM; this name comes from the "cold diffusion" paper discussed below.

Mainstream diffusion models handle continuous data, building the forward process with additive Gaussian noise:

\begin{equation}\boldsymbol{x}_t = \bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\end{equation}

The choice for $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ is the normal distribution $\mathcal{N}(\boldsymbol{x}_0;\bar{\boldsymbol{\mu}}(\boldsymbol{x}_t),\bar{\sigma}_t^2 \boldsymbol{I})$, and $\bar{\sigma}_t$ is generally not treated as a trainable parameter, so dropping the constant term gives

\begin{equation}-\log q(\boldsymbol{x}_0|\boldsymbol{x}_t) = \frac{1}{2\bar{\sigma}_t^2}\Vert\boldsymbol{x}_0 - \bar{\boldsymbol{\mu}}(\boldsymbol{x}_t)\Vert^2\end{equation}

Introducing the parameterization $\bar{\boldsymbol{\mu}}(\boldsymbol{x}_t) = \frac{1}{\bar{\alpha}_t}\left(\boldsymbol{x}_t - \bar{\beta}_t \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\right)$ and combining it with $\boldsymbol{x}_t = \bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}$ gives

\begin{equation}-\log q(\boldsymbol{x}_0|\boldsymbol{x}_t) = \frac{\bar{\beta}_t^2}{2\bar{\sigma}_t^2\bar{\alpha}_t^2}\left\Vert\boldsymbol{\varepsilon} - \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}, t)\right\Vert^2\end{equation}

Experiments show that dropping the leading coefficient works better, so the final training objective is generally $\Vert\boldsymbol{\varepsilon} - \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}, t)\Vert^2$. As for the choice of $\bar{\sigma}_t$ during sampling, one can refer to Generative Diffusion Models Explained (Part 7): Optimal Diffusion Variance Estimation (Part I) and Generative Diffusion Models Explained (Part 8): Optimal Diffusion Variance Estimation (Part II).

Finally, for $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0)$ we have

\begin{equation}\begin{aligned}\boldsymbol{x}_{t-1} =&\, \bar{\alpha}_{t-1} \boldsymbol{x}_0 + \bar{\beta}_{t-1} \boldsymbol{\varepsilon}\\ \sim&\, \bar{\alpha}_{t-1} \boldsymbol{x}_0 + \sqrt{\bar{\beta}_{t-1}^2 - \sigma_t^2}\boldsymbol{\varepsilon}_1 + \sigma_t\boldsymbol{\varepsilon}_2\end{aligned} \,\,,\quad \boldsymbol{\varepsilon},\boldsymbol{\varepsilon}_1,\boldsymbol{\varepsilon}_2\sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\end{equation}

Solving for $\boldsymbol{\varepsilon} = \left.(\boldsymbol{x}_t - \bar{\alpha}_t \boldsymbol{x}_0)\right/ \bar{\beta}_t$ from $\boldsymbol{x}_t = \bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}$ and substituting it in place of $\boldsymbol{\varepsilon}_1$, we eventually obtain the general form of $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0)$ as

\begin{equation}\boldsymbol{x}_{t-1} = \bar{\alpha}_{t-1} \boldsymbol{x}_0 + \sqrt{\bar{\beta}_{t-1}^2 - \sigma_t^2}\frac{\boldsymbol{x}_t - \bar{\alpha}_t \boldsymbol{x}_0}{\bar{\beta}_t} + \sigma_t\boldsymbol{\varepsilon},\quad\boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\end{equation}

while $\hat{\boldsymbol{x}}_0\sim p(\boldsymbol{x}_0|\boldsymbol{x}_t)$ implies

\begin{equation}\hat{\boldsymbol{x}}_0 = \bar{\boldsymbol{\mu}}(\boldsymbol{x}_t) + \bar{\sigma}_t \boldsymbol{\varepsilon} = \frac{1}{\bar{\alpha}_t}\left(\boldsymbol{x}_t - \bar{\beta}_t \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\right) + \bar{\sigma}_t \boldsymbol{\varepsilon}\end{equation}

Combining the two expressions above gives the most general form of the reverse process for mainstream diffusion models, where DDPM takes $\bar{\sigma}_t=0,\sigma_t = \frac{\bar{\beta}_{t-1}\beta_t}{\bar{\beta}_t}$, DDIM takes $\bar{\sigma}_t=0,\sigma_t = 0$, and Analytical-DPM instead re-estimates the optimal nonzero $\bar{\sigma}_t$.

Cold Diffusion

Next, we show that the "Cold Diffusion" introduced in Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise is also a special case of UDM. Cold Diffusion also deals with continuous data, and as the title of the paper suggests, it emphasizes constructing the forward process using arbitrary (noise-free) transformations. As far as I know, this is the first paper to attempt a general forward process, and UDM drew considerable inspiration from it during its construction — my thanks to the original authors.

Cold Diffusion builds the forward process through a deterministic transformation $\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0)$. To make the subsequent analysis easier, we introduce a more general forward process:

\begin{equation}\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0) + \sigma \boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim q(\boldsymbol{\varepsilon})\end{equation}

Here the transformation $\boldsymbol{\mathcal{F}}$ can be any kind of corruption applied to the original data — for images this could include blurring, masking, pooling, etc. If a deterministic transformation is desired, one simply lets $\sigma\to 0$ afterward.

Next, $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ is chosen as a normal distribution measured by the $l_1$ norm, i.e.

\begin{equation}q(\boldsymbol{x}_0|\boldsymbol{x}_t) = \frac{e^{-\left.\Vert\boldsymbol{x}_0 - \boldsymbol{\mathcal{G}}_t(\boldsymbol{x}_t)\Vert_1\right/\tau}}{Z(\tau)}\end{equation}

where $Z(\tau)=\int e^{-\left.\Vert\boldsymbol{x}_0 - \boldsymbol{\mathcal{G}}_t(\boldsymbol{x}_t)\Vert_1\right/\tau}d\boldsymbol{x}_0$ is the normalizing factor. Taking $\tau$ to be a fixed value makes $Z(\tau)$ a constant; dropping the constant term gives $-\log q(\boldsymbol{x}_0|\boldsymbol{x}_t)\propto\Vert\boldsymbol{x}_0 - \boldsymbol{\mathcal{G}}_t(\boldsymbol{x}_t)\Vert_1$, and combining this with $\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0)$, we get that the training objective is to minimize

\begin{equation}\Vert\boldsymbol{x}_0 - \boldsymbol{\mathcal{G}}_t(\boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0))\Vert_1\end{equation}

In the reverse process, Cold Diffusion simply ignores the variance of $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ (i.e., sets $\tau\to 0$), which gives $\hat{\boldsymbol{x}}_0 = \boldsymbol{\mathcal{G}}_t(\boldsymbol{x}_t)$. If $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0)$ directly takes the baseline choice $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_0)$, i.e. $\boldsymbol{x}_{t-1} = \boldsymbol{\mathcal{F}}_{t-1}(\boldsymbol{x}_0) + \sigma \boldsymbol{\varepsilon}$, then substituting into $\hat{\boldsymbol{x}}_0$ and taking the limit of $\sigma\to 0$ gives

\begin{equation}\hat{\boldsymbol{x}}_0=\boldsymbol{\mathcal{G}}_t(\boldsymbol{x}_t),\quad \boldsymbol{x}_{t-1} = \boldsymbol{\mathcal{F}}_{t-1}(\hat{\boldsymbol{x}}_0)\end{equation}

This is the "Naive Sampling" of the original paper. And if instead we solve for $\boldsymbol{\varepsilon} = \left.(\boldsymbol{x}_t - \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0))\right/\sigma$ from $\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0) + \sigma \boldsymbol{\varepsilon}$ and substitute into $\boldsymbol{x}_{t-1} = \boldsymbol{\mathcal{F}}_{t-1}(\boldsymbol{x}_0) + \sigma \boldsymbol{\varepsilon}$, we get

\begin{equation}\hat{\boldsymbol{x}}_0=\boldsymbol{\mathcal{G}}_t(\boldsymbol{x}_t),\quad \boldsymbol{x}_{t-1} = \boldsymbol{x}_t + \boldsymbol{\mathcal{F}}_{t-1}(\hat{\boldsymbol{x}}_0) - \boldsymbol{\mathcal{F}}_t(\hat{\boldsymbol{x}}_0)\end{equation}

This is the "Improved Sampling" of the original paper.

Overall, Cold Diffusion was the first to successfully realize a forward process based on general transformations, but because it places so much emphasis on being "without noise," it has an inherent, theoretically unavoidable flaw. For instance, for $w\times w\times 3$ image data, when Cold Diffusion implements the forward process using a blurring operation, the final result is equivalent to a $3$-dimensional vector. Since Cold Diffusion's reverse process is also deterministic, this means Cold Diffusion is using a deterministic transformation to turn a $3w^2$-dimensional image into a $3$-dimensional one, and then using another deterministic transformation to reconstruct a $3$-dimensional image back into $3w^2$ dimensions. There is necessarily severe information loss somewhere in this process, which inevitably limits the sharpness of the reconstruction, and consequently the sharpness of the generated results.

To solve this problem, one cannot refuse the presence of noise in either the forward or the reverse process. This is because noise implies uncertainty, uncertainty implies "one-to-many" mappings, and "one-to-many" mappings imply that the forward process is allowed to be "many-to-one" — that is, information loss is permitted. In fact, Cold Diffusion itself already recognized that a $3$-dimensional vector is inadequate for generating full $3w^2$-dimensional data: during generation it actually adds a slight amount of $3w^2$-dimensional random noise to this $3$-dimensional vector, and experiments show that this operation improves the generation quality. And this operation is roughly equivalent to the forward process of $\sigma > 0$.

Editing Models

Both examples above deal with continuous data, but as we mentioned, UDM in principle does not restrict the data type. In this section we introduce a discrete example, which shows that text generation models based on editing operations can also, in essence, be seen as special cases of UDM.

For simplicity, let's consider generating fixed-length sentences of length $l$, such as five-character or seven-character classical Chinese poetry lines. Variable-length sentences are not impossible, just somewhat more complex in detail. Then, we define the forward process $\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon})$ as "random replacement," namely:

Randomly select $t$ tokens in the sentence and randomly replace them with other tokens.

where $t\leq l$; when $t=l$, $\boldsymbol{x}_t$ becomes a sequence of $l$ completely randomly combined tokens.

Here $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ is the model that predicts the original sequence from the randomly replaced sequence; this can be implemented with either an autoregressive or non-autoregressive model, using cross-entropy as the loss function. Note that $\boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon})$ is necessarily non-invertible with respect to the noise (i.e., given $\boldsymbol{x}_0$ and $\boldsymbol{x}_t$, there is more than one way to go from $\boldsymbol{x}_0$ to $\boldsymbol{x}_t$), so we can only use the baseline choice $p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_t, \boldsymbol{x}_0)=p(\boldsymbol{x}_{t-1}|\boldsymbol{x}_0)$, which means the generation process is:

1. Randomly select $l$ tokens as the initial $\boldsymbol{x}_l$;
2. Predict $\hat{\boldsymbol{x}}_0$ from $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$;
3. Randomly select $t-1$ tokens in $\hat{\boldsymbol{x}}_0$ and randomly replace them with other tokens, giving $\boldsymbol{x}_{t-1}$;
4. Repeat steps 2 and 3 until the final $\boldsymbol{x}_0$ is obtained.

However, an algorithm like this doesn't work very well, because the predictive progress made in step 2 often gets largely "destroyed" by the random replacement in step 3 — a bit like "reverting back to square one overnight." To improve on this, we need a better sampling scheme, which requires $\boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon})$ to be invertible with respect to the noise, meaning that from the given $\boldsymbol{x}_0$ and $\boldsymbol{x}_t$ we can tell exactly what transformation was applied. To this end, we define the forward process as:

Randomly select $t$ tokens in the sentence and randomly replace them with different tokens.

The difference from before is that during random replacement, the original token must be replaced by a token that differs from it; otherwise, we might sample the same token again. With this restriction in place, we can directly compare $\boldsymbol{x}_0$ and $\boldsymbol{x}_t$ to see what was changed, and thereby replace the random replacement of step 3 with a transformation from $\hat{\boldsymbol{x}}_0$ to $\boldsymbol{x}_t$:

1. Randomly select $l$ tokens as the initial $\boldsymbol{x}_l$;
2. Predict $\hat{\boldsymbol{x}}_0$ from $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$, requiring that $\hat{\boldsymbol{x}}_0$ and $\boldsymbol{x}_t$ differ in $t$ tokens (this is easier to implement with a non-autoregressive model);
3. Randomly select one of the tokens in $\boldsymbol{x}_t$ that differs from $\hat{\boldsymbol{x}}_0$, and replace it with the corresponding token from $\hat{\boldsymbol{x}}_0$, giving $\boldsymbol{x}_{t-1}$;
4. Repeat steps 2 and 3 until the final $\boldsymbol{x}_0$ is obtained.

In this way, the valid part of each prediction $\hat{\boldsymbol{x}}_0$ (the part where $\hat{\boldsymbol{x}}_0$ agrees with $\boldsymbol{x}_t$) is always preserved, and $\boldsymbol{x}_{t-1}$ differs from $\boldsymbol{x}_t$ by only a single token, so the generation process proceeds as a stable, progressive refinement. The difference from an ordinary autoregressive model is that the constraint of generating strictly left-to-right has been removed.

Masked Models

If readers are still a bit fuzzy on the model above, let me offer a simpler example to aid understanding. Again consider generating fixed-length sentences of length $l$, and define the forward process $\boldsymbol{x}_t = \boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon})$ as "random masking," namely:

Randomly select $t$ tokens in the sentence and replace them with [MASK].

where $t\leq l$; when $t=l$, $\boldsymbol{x}_t$ consists of $l$ [MASK] tokens.

Here $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$ is the model that predicts the original sequence from the masked sequence, typically implemented with a BERT-like MLM model (a non-autoregressive model), with cross-entropy as the loss function. The baseline generation process is as follows:

The generation process is:

1. Start with $l$ [MASK] tokens as the initial $\boldsymbol{x}_l$;
2. Sample $\hat{\boldsymbol{x}}_0$ from $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$;
3. Randomly select $t-1$ tokens in $\hat{\boldsymbol{x}}_0$ and replace them with [MASK], giving $\boldsymbol{x}_{t-1}$;
4. Repeat steps 2 and 3 until the final $\boldsymbol{x}_0$ is obtained.

Note that here $\boldsymbol{\mathcal{F}}_t(\boldsymbol{x}_0,\boldsymbol{\varepsilon})$ is invertible with respect to the noise — that is, from the given $\boldsymbol{x}_0$ and $\boldsymbol{x}_t$ we can tell exactly what transformation was applied (i.e., which tokens were replaced with [MASK]). We can therefore construct an improved generation process:

1. Start with $l$ [MASK] tokens as the initial $\boldsymbol{x}_l$;
2. Sample $\hat{\boldsymbol{x}}_0$ from $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$, noting that we only need to sample the tokens that were originally [MASK]; tokens that were not [MASK] remain unchanged;
3. Randomly select $t-1$ positions out of the $t$ [MASK] positions in the original $\boldsymbol{x}_t$, and replace the tokens at these positions in $\hat{\boldsymbol{x}}_0$ with [MASK], giving $\boldsymbol{x}_{t-1}$;
4. Repeat steps 2 and 3 until the final $\boldsymbol{x}_0$ is obtained.

Of course, steps 2 and 3 can actually be merged into a more direct single step:

2 & 3. Randomly select $1$ positions out of the $t$ [MASK] positions in $\boldsymbol{x}_t$, and sample a token to fill in at each position according to the corresponding probabilities from $q(\boldsymbol{x}_0|\boldsymbol{x}_t)$, giving $\boldsymbol{x}_{t-1}$;

This is almost identical to Gibbs sampling based on an MLM model (see Sampled Text Series ⋅ (3): BERT-based Text Sampling). From these two examples — the "editing model" and the "masked model" — we should get a rough sense that many "gradual generation" models can be re-expressed within the UDM framework. Or, conversely, any gradual generation scheme we can think of can be probabilistically formulated by attempting to fit it into the UDM framework.

Encoding Models

The forward processes we've discussed so far all lack trainable parameters — that is, they're all pre-designed procedures. But this isn't actually necessary. We can generalize DDPM's diffusion process to

\begin{equation}\boldsymbol{x}_t = \bar{\alpha}_t \boldsymbol{\mathcal{F}}(\boldsymbol{x}_0) + \bar{\beta}_t \boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I})\end{equation}

where $\boldsymbol{\mathcal{F}}(\boldsymbol{x}_0)$ is an encoding model applied to $\boldsymbol{x}_0$, which may have trainable parameters. In this case, the training objective becomes

\begin{equation}-\log q(\boldsymbol{x}_0|\boldsymbol{x}_t) = -\log q(\boldsymbol{x}_0|\bar{\alpha}_t\boldsymbol{\mathcal{F}}(\boldsymbol{x}_0) + \bar{\beta}_t \boldsymbol{\varepsilon})\end{equation}

except that here $\boldsymbol{\mathcal{F}}$ also has trainable parameters. As for the reverse process, it works similarly, except that once $\hat{\boldsymbol{x}}_0\sim q(\boldsymbol{x}_0|\boldsymbol{x}_1)$ is sampled it is directly returned as $\hat{\boldsymbol{x}}_0$. In particular, because of this additional encoding model $\boldsymbol{\mathcal{F}}$, the input $\boldsymbol{x}_0$ can be either discrete or continuous data — this provides a VAE-like way of encoding the data distribution into a normal distribution over latent variables.

Summary

This post has mainly applied the Unified Diffusion Model (UDM) framework constructed in the previous post to derive several concrete examples, including mainstream diffusion models, Cold Diffusion, text-editing generation, and encoding models.