Generative Diffusion Models Explained (V): The SDE Formulation of the General Framework
When I wrote the first article of the generative diffusion models series, a reader in the comments already recommended Dr. Yang Song's paper Score-Based Generative Modeling through Stochastic Differential Equations. This paper builds a fairly general theoretical framework for generative diffusion models, tying together DDPM, SDEs, ODEs, and many other results. It's certainly a good paper, but not one that's particularly friendly to beginners: it directly makes heavy use of stochastic differential equations (SDEs), the Fokker-Planck equation, score matching, and other substantial machinery, so the barrier to entry is quite high.
However, having accumulated some groundwork over the previous four articles, we're now in a position to try to learn from this paper. In what follows, I'll try to start from as little theoretical background as possible and reproduce, as faithfully as I can, the derivations in the original paper.
Stochastic Differentials
In DDPM, the diffusion process is divided into a fixed number of $T$ steps — to use the analogy from Generative Diffusion Models Explained (I): DDPM = Demolishing Buildings + Building Buildings, both "demolition" and "construction" are pre-divided into $T$ steps, and this division is fairly arbitrary. In fact, the real "demolition" and "construction" processes shouldn't have such artificially imposed steps — we can instead think of them as a transformation that unfolds continuously in time, which can be described by a stochastic differential equation (SDE). more
To this end, we describe the forward process ("demolition") with the following SDE:
\begin{equation}d\boldsymbol{x} = \boldsymbol{f}_t(\boldsymbol{x}) dt + g_t d\boldsymbol{w}\label{eq:sde-forward}\end{equation}
I suspect many readers are unfamiliar with SDEs — I myself only dabbled in them briefly during my master's studies, so I only know the basics. But that's fine; we just need to treat it as the limit, as $\Delta t\to 0$, of the following discretized form:
\begin{equation}\boldsymbol{x}_{t+\Delta t} - \boldsymbol{x}_t = \boldsymbol{f}_t(\boldsymbol{x}_t) \Delta t + g_t \sqrt{\Delta t}\boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I})\label{eq:sde-discrete}\end{equation}
To put it more plainly: suppose demolishing the building takes $1$ days. Then demolition is the process of $\boldsymbol{x}$ evolving from $t=0$ to $t=1$, and each small step of change can be described by the equation above. As for the time interval $\Delta t$, we don't impose any particular restriction on it — smaller $\Delta t$ simply gives a better approximation to the original SDE. If we take $\Delta t=0.001$, this corresponds to the original $T=1000$; if we take $\Delta t = 0.01$, it corresponds to $T=100$, and so on. In other words, from the continuous-time SDE perspective, different choices of $T$ are simply different levels of discretization of the SDE, and they will automatically lead to similar results. We don't need to fix $T$ in advance — instead, we choose an appropriate $T$ for numerical computation based on the required precision in practice.
So the essential benefit of introducing the SDE formulation to describe diffusion models is that it "separates theoretical analysis from code implementation": we can use the mathematical tools of continuous-time SDEs to analyze the model, while in practice we only need to use any suitable discretization scheme to numerically solve the SDE.
For equation $\eqref{eq:sde-discrete}$, readers might reasonably wonder: why is the first term on the right-hand side of order $\mathcal{O}(\Delta t)$, while the second term is of order $\mathcal{O}(\sqrt{\Delta t})$? That is, why does the order of the stochastic term have to be higher than that of the deterministic term? This isn't so easy to explain, and it's actually one of the more confusing aspects of SDEs. Roughly speaking, it's because $\boldsymbol{\varepsilon}$ is always distributed as a standard normal. If the weight on the stochastic term were also $\mathcal{O}(\Delta t)$, then since the standard normal has mean $\boldsymbol{0}$ and covariance $ \boldsymbol{I}$, nearby random effects would cancel each other out; the weight needs to be amplified to $\mathcal{O}(\sqrt{\Delta t})$ for the stochastic effect to actually show up in the long-run result.
The Reverse Equation
In the language of probability, equation $\eqref{eq:sde-discrete}$ means the conditional probability is
\begin{equation}\begin{aligned} p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t) =&\, \mathcal{N}\left(\boldsymbol{x}_{t+\Delta t};\boldsymbol{x}_t + \boldsymbol{f}_t(\boldsymbol{x}_t) \Delta t, g_t^2\Delta t \,\boldsymbol{I}\right)\\ \propto&\, \exp\left(-\frac{\Vert\boldsymbol{x}_{t+\Delta t} - \boldsymbol{x}_t - \boldsymbol{f}_t(\boldsymbol{x}_t) \Delta t\Vert^2}{2 g_t^2\Delta t}\right) \end{aligned}\label{eq:sde-proba}\end{equation}
For simplicity, I've omitted the (irrelevant) normalization factor here. Following the DDPM philosophy, our ultimate goal is to learn "construction" from the "demolition" process, i.e., to obtain $p(\boldsymbol{x}_t|\boldsymbol{x}_{t+\Delta t})$. To do this, just as in Generative Diffusion Models Explained (III): DDPM = Bayes + Denoising, we use Bayes' theorem:
\begin{equation}\begin{aligned} p(\boldsymbol{x}_t|\boldsymbol{x}_{t+\Delta t}) =&\, \frac{p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t)p(\boldsymbol{x}_t)}{p(\boldsymbol{x}_{t+\Delta t})} = p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t) \exp\left(\log p(\boldsymbol{x}_t) - \log p(\boldsymbol{x}_{t+\Delta t})\right)\\ \propto&\, \exp\left(-\frac{\Vert\boldsymbol{x}_{t+\Delta t} - \boldsymbol{x}_t - \boldsymbol{f}_t(\boldsymbol{x}_t) \Delta t\Vert^2}{2 g_t^2\Delta t} + \log p(\boldsymbol{x}_t) - \log p(\boldsymbol{x}_{t+\Delta t})\right) \end{aligned}\label{eq:bayes-dt}\end{equation}
It's not hard to see that when $\Delta t$ is sufficiently small, $\boldsymbol{x}_{t+\Delta t}$ can be appreciably nonzero only when it's sufficiently close to $\boldsymbol{x}_t$, and conversely $p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t)$ can be appreciably nonzero only in this same regime — so $p(\boldsymbol{x}_t|\boldsymbol{x}_{t+\Delta t})$ is nonnegligible only under the same condition. Hence, we only need to carry out the approximate analysis for the regime where $\boldsymbol{x}_{t+\Delta t}$ and $\boldsymbol{x}_t$ are close to each other, using a Taylor expansion:
\begin{equation}\log p(\boldsymbol{x}_{t+\Delta t})\approx \log p(\boldsymbol{x}_t) + (\boldsymbol{x}_{t+\Delta t} - \boldsymbol{x}_t)\cdot \nabla_{\boldsymbol{x}_t}\log p(\boldsymbol{x}_t) + \Delta t \frac{\partial}{\partial t}\log p(\boldsymbol{x}_t)\end{equation}
Be careful not to drop the $\frac{\partial}{\partial t}$ term, because $p(\boldsymbol{x}_t)$ is actually "the probability density that the random variable at time $t$ equals $\boldsymbol{x}_t$," while $p(\boldsymbol{x}_{t+\Delta t})$ is actually "the probability density that the random variable at time $t+\Delta t$ equals $\boldsymbol{x}_{t+\Delta t}$." In other words, $p(\boldsymbol{x}_t)$ is simultaneously a function of both $t$ and $\boldsymbol{x}_t$, so we need an extra partial-derivative term with respect to $t$. Substituting this into equation $\eqref{eq:bayes-dt}$ and completing the square, we get
\begin{equation}p(\boldsymbol{x}_t|\boldsymbol{x}_{t+\Delta t}) \propto \exp\left(-\frac{\Vert\boldsymbol{x}_{t+\Delta t} - \boldsymbol{x}_t - \left[\boldsymbol{f}_t(\boldsymbol{x}_t) - g_t^2\nabla_{\boldsymbol{x}_t}\log p(\boldsymbol{x}_t) \right]\Delta t\Vert^2}{2 g_t^2\Delta t} + \mathcal{O}(\Delta t)\right)\end{equation}
When $\Delta t\to 0$, the term $\mathcal{O}(\Delta t)\to 0$ no longer matters, so
\begin{equation}\begin{aligned} p(\boldsymbol{x}_t|\boldsymbol{x}_{t+\Delta t}) \propto&\, \exp\left(-\frac{\Vert\boldsymbol{x}_{t+\Delta t} - \boldsymbol{x}_t - \left[\boldsymbol{f}_t(\boldsymbol{x}_t) - g_t^2\nabla_{\boldsymbol{x}_t}\log p(\boldsymbol{x}_t) \right]\Delta t\Vert^2}{2 g_t^2\Delta t}\right) \\ \approx&\,\exp\left(-\frac{\Vert \boldsymbol{x}_t - \boldsymbol{x}_{t+\Delta t} + \left[\boldsymbol{f}_{t+\Delta t}(\boldsymbol{x}_{t+\Delta t}) - g_{t+\Delta t}^2\nabla_{\boldsymbol{x}_{t+\Delta t}}\log p(\boldsymbol{x}_{t+\Delta t}) \right]\Delta t\Vert^2}{2 g_{t+\Delta t}^2\Delta t}\right) \end{aligned}\end{equation}
That is, $p(\boldsymbol{x}_t|\boldsymbol{x}_{t+\Delta t})$ is approximately a normal distribution with mean $\boldsymbol{x}_{t+\Delta t} - \left[\boldsymbol{f}_{t+\Delta t}(\boldsymbol{x}_{t+\Delta t}) - g_{t+\Delta t}^2\nabla_{\boldsymbol{x}_{t+\Delta t}}\log p(\boldsymbol{x}_{t+\Delta t}) \right]\Delta t$ and covariance $g_{t+\Delta t}^2\Delta t\,\boldsymbol{I}$. Taking the limit $\Delta t\to 0$, this corresponds to the SDE:
\begin{equation}d\boldsymbol{x} = \left[\boldsymbol{f}_t(\boldsymbol{x}) - g_t^2\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x}) \right] dt + g_t d\boldsymbol{w}\label{eq:reverse-sde}\end{equation}
This is the SDE for the reverse process, and it first appeared in Reverse-Time Diffusion Equation Models. Here we've deliberately marked the subscript $t$ on $p$ to emphasize that this is the distribution at time $t$.
Score Matching
Now that we have the reverse SDE $\eqref{eq:reverse-sde}$, if we further know $\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x})$, then we can use the discretization scheme
\begin{equation}\boldsymbol{x}_t - \boldsymbol{x}_{t+\Delta t} = - \left[\boldsymbol{f}_{t+\Delta t}(\boldsymbol{x}_{t+\Delta t}) - g_{t+\Delta t}^2\nabla_{\boldsymbol{x}_{t+\Delta t}}\log p(\boldsymbol{x}_{t+\Delta t}) \right]\Delta t - g_{t+\Delta t} \sqrt{\Delta t}\boldsymbol{\varepsilon}\label{eq:reverse-sde-discrete}\end{equation}
to carry out "construction" step by step [where $\boldsymbol{\varepsilon}\sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{I})$], thereby completing the construction of a generative diffusion model.
So how do we obtain $\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x})$? At time $t$, the quantity $p_t(\boldsymbol{x})$ is precisely the earlier $p(\boldsymbol{x}_t)$, meaning the marginal distribution at time $t$. In practice, we generally design models such that $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$ has a closed-form solution, which means that
\begin{equation}\small p(\boldsymbol{x}_t|\boldsymbol{x}_0) = \lim_{\Delta t\to 0}\int\cdots\iint p(\boldsymbol{x}_t|\boldsymbol{x}_{t-\Delta t})p(\boldsymbol{x}_{t-\Delta t}|\boldsymbol{x}_{t-2\Delta t})\cdots p(\boldsymbol{x}_{\Delta t}|\boldsymbol{x}_0) d\boldsymbol{x}_{t-\Delta t} d\boldsymbol{x}_{t-2\Delta t}\cdots d\boldsymbol{x}_{\Delta t}\end{equation}
can be computed directly — for instance, when $\boldsymbol{f}_t(\boldsymbol{x})$ is a linear function of $\boldsymbol{x}$, $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$ can be solved analytically. Under this assumption, we have
\begin{equation}p(\boldsymbol{x}_t) = \int p(\boldsymbol{x}_t|\boldsymbol{x}_0)\tilde{p}(\boldsymbol{x}_0)d\boldsymbol{x}_0=\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]\end{equation}
hence
\begin{equation}\nabla_{\boldsymbol{x}_t}\log p(\boldsymbol{x}_t) = \frac{\mathbb{E}_{\boldsymbol{x}_0}\left[\nabla_{\boldsymbol{x}_t} p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]}{\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]} = \frac{\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]}{\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]}\end{equation}
Notice that the final expression takes the form of "a weighted average of $\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)$." Since we've assumed $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$ has a closed-form solution, the above expression can, in principle, be estimated directly. However, it involves averaging over the entire training set $\boldsymbol{x}_0$, which is both computationally expensive and prone to poor generalization. So instead, we want to use a neural network to learn a function $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ that can directly compute $\nabla_{\boldsymbol{x}_t}\log p(\boldsymbol{x}_t)$.
Many readers will already be familiar with the following result (or it's not hard to derive it yourself):
\begin{equation}\mathbb{E}[\boldsymbol{x}] = \mathop{\text{argmin}}_{\boldsymbol{\mu}}\mathbb{E}_{\boldsymbol{x}}\left[\Vert \boldsymbol{\mu} - \boldsymbol{x}\Vert^2\right]\end{equation}
That is, to make $\boldsymbol{\mu}$ equal to the mean of $\boldsymbol{x}$, we only need to minimize the mean of $\Vert \boldsymbol{\mu} - \boldsymbol{x}\Vert^2$. By the same logic, to make $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ equal to the weighted average of $\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)$ [namely $\nabla_{\boldsymbol{x}_t}\log p(\boldsymbol{x}_t)$], we only need to minimize the weighted average of $\left\Vert \boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right\Vert^2$, i.e.,
\begin{equation} \frac{\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\left\Vert \boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right\Vert^2\right]}{\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]}\end{equation}
The $\mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right]$ in the denominator only serves to adjust the weighting of the loss; for simplicity we can drop it without affecting the optimal solution. Finally, integrating over $\boldsymbol{x}_t$ (which amounts to minimizing the above loss for every value of $\boldsymbol{x}_t$), we obtain the final loss function
\begin{equation}\begin{aligned}&\,\int \mathbb{E}_{\boldsymbol{x}_0}\left[p(\boldsymbol{x}_t|\boldsymbol{x}_0)\left\Vert \boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right\Vert^2\right] d\boldsymbol{x}_t \\ =&\, \mathbb{E}_{\boldsymbol{x}_0,\boldsymbol{x}_t \sim p(\boldsymbol{x}_t|\boldsymbol{x}_0)\tilde{p}(\boldsymbol{x}_0)}\left[\left\Vert \boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0)\right\Vert^2\right] \end{aligned}\label{eq:score-match}\end{equation}
This is the loss function for "(conditional) score matching." The closed-form solution for the denoising autoencoder that we derived earlier in From Denoising Autoencoders to Generative Models is in fact a special case of this. The earliest origin of score matching can be traced back to the 2005 paper Estimation of Non-Normalized Statistical Models by Score Matching, while for conditional score matching, the earliest reference I've been able to trace is the 2011 paper A Connection Between Score Matching and Denoising Autoencoders.
That said, although the result here matches score matching, in this section's derivation we've actually done away with the notion of "score" entirely — the answer emerges naturally and directly from the objective. I think this way of presenting it is more illuminating, and I hope this derivation makes score matching easier to understand.
Working Backwards from the Result
At this point, we've established the general procedure for building a generative diffusion model:
1. Define "demolition" (the forward process) via a stochastic differential equation $\eqref{eq:sde-forward}$;
2. Find the expression for $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$;
3. Train $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ using the loss function $\eqref{eq:score-match}$ (score matching);
4. Replace $\nabla_{\boldsymbol{x}}\log p_t(\boldsymbol{x})$ in equation $\eqref{eq:reverse-sde}$ with $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ to complete "construction" (the reverse process).
When we see terms like SDE and differential equations, it's natural to feel a bit "intimidated," but fundamentally, the SDE is just a "front." Once we translate our understanding of the SDE into equations $\eqref{eq:sde-discrete}$ and $\eqref{eq:sde-proba}$, we can drop the SDE framing entirely, so conceptually this really isn't that difficult.
It's easy to see that defining a stochastic differential equation $\eqref{eq:sde-forward}$ is straightforward, but solving for $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$ given $\eqref{eq:sde-forward}$ is not. Most of the remaining content in the original paper is devoted to deriving and experimenting with two practically useful examples. However, since solving for $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$ is not easy, in my view, rather than defining $\eqref{eq:sde-forward}$ first and then solving for $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$, wouldn't it make more sense to follow the approach of DDIM — first define $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$, and then work backwards to find the corresponding SDE?
For example, let's first define
\begin{equation} p(\boldsymbol{x}_t|\boldsymbol{x}_0) = \mathcal{N}(\boldsymbol{x}_t; \bar{\alpha}_t \boldsymbol{x}_0,\bar{\beta}_t^2 \boldsymbol{I})\end{equation}
and, without loss of generality, assume the starting point is $t=0$ and the ending point is $t=1$. Then the boundary conditions that $\bar{\alpha}_t,\bar{\beta}_t$ must satisfy are
\begin{equation} \bar{\alpha}_0 = 1,\quad \bar{\alpha}_1 = 0,\quad \bar{\beta}_0 = 0,\quad \bar{\beta}_1 = 1\end{equation}
Of course, in principle these boundary conditions just need to hold approximately — they don't have to be exactly satisfied. For instance, in the previous article we analyzed that DDPM corresponds to choosing $\bar{\alpha}_t = e^{-5t^2}$, whose value at $t=1$ is $e^{-5}\approx 0$.
Given $p(\boldsymbol{x}_t|\boldsymbol{x}_0)$, to work backwards to $\eqref{eq:sde-forward}$, we essentially need to solve for $p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t)$, which must satisfy
\begin{equation} p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_0) = \int p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t) p(\boldsymbol{x}_t|\boldsymbol{x}_0) d\boldsymbol{x}_t\end{equation}
Let's consider a linear solution, i.e.,
\begin{equation}d\boldsymbol{x} = f_t\boldsymbol{x} dt + g_t d\boldsymbol{w}\end{equation}
Just as in Generative Diffusion Models Explained (IV): DDIM = A Higher Vantage Point on DDPM, we write
$$\begin{array}{c|c|c} \hline \text{notation} & \text{meaning} & \text{sampling}\\ \hline p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_0) & \mathcal{N}(\boldsymbol{x}_t;\bar{\alpha}_{t+\Delta t} \boldsymbol{x}_0,\bar{\beta}_{t+\Delta t}^2 \boldsymbol{I}) & \boldsymbol{x}_{t+\Delta t} = \bar{\alpha}_{t+\Delta t} \boldsymbol{x}_0 + \bar{\beta}_{t+\Delta t} \boldsymbol{\varepsilon} \\ \hline p(\boldsymbol{x}_t|\boldsymbol{x}_0) & \mathcal{N}(\boldsymbol{x}_t;\bar{\alpha}_t \boldsymbol{x}_0,\bar{\beta}_t^2 \boldsymbol{I}) & \boldsymbol{x}_t = \bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}_1 \\ \hline p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t) & \mathcal{N}(\boldsymbol{x}_{t+\Delta t}; (1 + f_t\Delta t) \boldsymbol{x}_t, g_t^2 \Delta t\, \boldsymbol{I}) & \boldsymbol{x}_{t+\Delta t} = (1 + f_t\Delta t) \boldsymbol{x}_t + g_t\sqrt{\Delta t}\boldsymbol{\varepsilon}_2 \\ \hline {\begin{array}{c}\int p(\boldsymbol{x}_{t+\Delta t}|\boldsymbol{x}_t) \\ p(\boldsymbol{x}_t|\boldsymbol{x}_0) d\boldsymbol{x}_t\end{array}$$} & & {$$\begin{aligned}&\,\boldsymbol{x}_{t+\Delta t} \\ =&\, (1 + f_t\Delta t) \boldsymbol{x}_t + g_t\sqrt{\Delta t} \boldsymbol{\varepsilon}_2 \\ =&\, (1 + f_t\Delta t) (\bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}_1) + g_t\sqrt{\Delta t} \boldsymbol{\varepsilon}_2 \\ =&\, (1 + f_t\Delta t) \bar{\alpha}_t \boldsymbol{x}_0 + ((1 + f_t\Delta t)\bar{\beta}_t \boldsymbol{\varepsilon}_1 + g_t\sqrt{\Delta t} \boldsymbol{\varepsilon}_2) \\ \end{aligned}$$} \\
\hline
\end{array}
From this we obtain
\begin{equation}\begin{aligned} \bar{\alpha}_{t+\Delta t} =&\, (1 + f_t\Delta t) \bar{\alpha}_t \\ \bar{\beta}_{t+\Delta t}^2 =&\, (1 + f_t\Delta t)^2\bar{\beta}_t^2 + g_t^2\Delta t \end{aligned}\end{equation}
Setting $\Delta t\to 0$, we solve separately to get
\begin{equation} f_t = \frac{d}{dt} \left(\ln \bar{\alpha}_t\right) = \frac{1}{\bar{\alpha}_t}\frac{d\bar{\alpha}_t}{dt}, \quad g_t^2 = \bar{\alpha}_t^2 \frac{d}{dt}\left(\frac{\bar{\beta}_t^2}{\bar{\alpha}_t^2}\right) = 2\bar{\alpha}_t \bar{\beta}_t \frac{d}{dt}\left(\frac{\bar{\beta}_t}{\bar{\alpha}_t}\right)\end{equation}
Taking $\bar{\alpha}_t\equiv 1$ gives the result that is VE-SDE (Variance Exploding SDE) in the paper; taking $\bar{\alpha}_t^2 + \bar{\beta}_t^2=1$ gives VP-SDE (Variance Preserving SDE) from the original paper.
As for the loss function, we can now compute
\begin{equation}\nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t|\boldsymbol{x}_0) = -\frac{\boldsymbol{x}_t - \bar{\alpha}_t\boldsymbol{x}_0}{\bar{\beta}_t^2}=-\frac{\boldsymbol{\varepsilon}}{\bar{\beta}_t}\end{equation}
The second equality holds because $\boldsymbol{x}_t = \bar{\alpha}_t\boldsymbol{x}_0 + \bar{\beta}_t\boldsymbol{\varepsilon}$; to align with previous results, we set $\boldsymbol{s}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) = -\frac{\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)}{\bar{\beta}_t}$, so equation $\eqref{eq:score-match}$ becomes
\begin{equation}\frac{1}{\bar{\beta}_t^2}\mathbb{E}_{\boldsymbol{x}_0\sim \tilde{p}(\boldsymbol{x}_0),\boldsymbol{\varepsilon}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})}\left[\left\Vert \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\bar{\alpha}_t\boldsymbol{x}_0 + \bar{\beta}_t\boldsymbol{\varepsilon}, t) - \boldsymbol{\varepsilon}\right\Vert^2\right]\end{equation}
Dropping the coefficient, this is exactly the DDPM loss function. And if we replace $\nabla_{\boldsymbol{x}_{t+\Delta t}}\log p(\boldsymbol{x}_{t+\Delta t})$ in equation $\eqref{eq:reverse-sde-discrete}$ with $-\frac{\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_{t+\Delta t}, t+\Delta t)}{\bar{\beta}_{t+\Delta t}}$, the result shares the same first-order approximation as DDPM's sampling process (meaning the two are equivalent when $\Delta t\to 0$).
Summary
This article has mainly introduced the general framework Dr. Yang Song established for understanding diffusion models via SDEs, deriving results such as the reverse SDE and score matching using as intuitive a language as possible, and offering my own thoughts on how to solve the associated equations.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.