Diffusion Models for Generation, Chatting Casually (Part 23): Signal-to-Noise Ratio and Large-Image Generation (Part 2)
In the previous post, Diffusion Models for Generation, Chatting Casually (Part 22): Signal-to-Noise Ratio and Large-Image Generation (Part 1), we introduced a way to improve the noise schedule by aligning the signal-to-noise ratio with that of low-resolution training, thereby improving diffusion models that are trained directly in pixel space for high-resolution image generation (i.e., "large image" generation). This post also has signal-to-noise ratio and large-image generation as its protagonists, but it achieves something even more astonishing: taking a diffusion model trained on low-resolution images and using it directly for high-resolution image generation, with no additional training, while matching both the quality and inference cost of a model trained directly at the higher resolution!
This work comes from the recent paper Upsample Guidance: Scale Up Diffusion Models without Training, which cleverly uses upsampling of the low-resolution model as a guidance signal, combined with the shift-invariance of CNNs with respect to texture detail, to successfully achieve training-free high-resolution image generation.
A Conceptual Exploration
As we know, the training objective of diffusion models is denoising (the first "D" in DDPM). Intuitively, denoising should be a resolution-agnostic task — in other words, ideally, a denoising model trained on low-resolution images should also work for denoising high-resolution images, and consequently a low-resolution diffusion model should be directly usable for high-resolution image generation. more
Is it really that ideal? I tried it out with a 128×128 face-image (CelebA-HQ) diffusion model I had previously trained myself — that is, I directly used it as if it were a 256×256 model for inference. The generated results looked like this:
Generation results from using a 128-resolution diffusion model as if it were a 256-resolution model
As you can see, the generated results have two notable characteristics:
1. The generated results are no longer face images at all, showing that a denoising model trained at 128×128 cannot be directly used as a 256×256 model;
2. Although the generated results are not satisfactory, they are quite sharp, with no obvious blurring or checkerboard artifacts, and they even retain some facial texture detail.
We know that simply enlarging (upsampling) a small image is the most basic form of large-image generation model, but depending on the upsampling algorithm, the enlarged image typically suffers from blurring or checkerboard artifacts — that is, a lack of sufficient texture detail. This naturally suggests a somewhat wild idea: since upsampling a small image lacks detail, while treating the small-image model directly as a large-image model preserves some detail, could we use the latter to supplement the former with detail?
This is precisely the core idea behind the method proposed in the paper.
Mathematical Formulation
In this section, let's restate the idea formally, to see what the next step should be.
First, let's fix notation. Suppose our target image resolution is $w\times h$ and the training image resolution is $w/s\times h/s$, so below, $\boldsymbol{x},\boldsymbol{\varepsilon}$ is always of size $w\times h\times 3$ (with an additional channel dimension for images), while $\boldsymbol{x}^{\text{low}},\boldsymbol{\varepsilon}^{\text{low}}$ is always of size $w\times h\times 3$. $\mathcal{D}$ denotes the downsampling operator that average-pools from resolution $w\times h$ down to $w/s\times h/s$, while $\mathcal{U}$ denotes the upsampling operator that nearest-neighbor interpolates (i.e., simply repeats) from resolution $w/s\times h/s$ up to $w\times h$.
As we know, a diffusion model requires a trained denoising model $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$. Taking DDPM as an example (here we use the form from Diffusion Models for Generation, Chatting Casually (Part 3): DDPM = Bayes + Denoising, which is basically consistent with the mainstream formulation), its inference formula is
\begin{equation}\boldsymbol{x}_{t-1} = \frac{1}{\alpha_t}\left(\boldsymbol{x}_t - \frac{\beta_t^2}{\bar{\beta}_t}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\right) + \sigma_t \boldsymbol{\varepsilon},\quad \boldsymbol{\varepsilon}\sim\mathcal{N}(\boldsymbol{0}, \boldsymbol{I})\end{equation}
where the mainstream choice for $\sigma_t$ is either $\frac{\bar{\beta}_{t-1}\beta_t}{\bar{\beta}_t}$ or $\beta_t$. But now we don't have a $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ trained at resolution $w\times h$ — we only have a $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t^{\text{low}}, t)$ trained at resolution $w/s\times h/s$.
From experience, shrinking a large image and then enlarging it back, although lossy, still produces a reasonably good approximation of the original image. This inspires us to construct a "main term" for the denoising model in a similar way. Specifically, in order to denoise an image of size $w\times h$, we can first shrink it (downsample via average pooling) to $w/s\times h/s$, feed it into the denoising model trained at resolution $w/s\times h/s$, and then enlarge (upsample) the denoised result back to $w\times h$. While this won't be the ideal denoising result, it should already constitute a main term of the ideal result.
Next, in the previous section we demonstrated that directly using a denoising model trained at low resolution as a high-resolution model retains some texture detail. So we can view the completely unmodified $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ as constituting a secondary term that captures detail. If we can find a way to combine these two terms — the main term and the secondary term — we might obtain a sufficiently good approximation to the precise denoising model, and thereby achieve training-free high-resolution diffusion generation.
Calling on SNR Again
Now let's discuss the main term. First, let's be clear: this work is not about retraining a new high-resolution model, but about reusing the original low-resolution model on high-resolution inputs. So the noise schedule is still the original $\bar{\alpha}_t,\bar{\beta}_t$, and we can therefore assume that we still have
\begin{equation}\boldsymbol{x}_t = \bar{\alpha}_t \boldsymbol{x}_0 + \bar{\beta}_t \boldsymbol{\varepsilon}\end{equation}
where $\boldsymbol{\varepsilon}$ is a vector drawn from the standard normal distribution. As discussed in the previous section, the main term requires downsampling first and then denoising. Let $\mathcal{D}$ denote the average-pooling operation that downsamples to $w/s\times h/s$; then we have
\begin{equation}\mathcal{D}[\boldsymbol{x}_t] = \bar{\alpha}_t \mathcal{D}[\boldsymbol{x}_0] + \frac{\bar{\beta}_t}{s} \boldsymbol{\varepsilon}\label{eq:dx}\end{equation}
Here, "equal" means "equal in distribution." In the previous post, we introduced the signal-to-noise ratio $SNR(t)=\frac{\bar{\alpha}_t^2}{\bar{\beta}_t^2}$, from which we can see that the signal-to-noise ratio of $\boldsymbol{x}_t$ is $\frac{\bar{\alpha}_t^2}{\bar{\beta}_t^2}$, while that of $\mathcal{D}[\boldsymbol{x}_t]$ is $\frac{s^2\bar{\alpha}_t^2}{\bar{\beta}_t^2}$. Given the setup of this post, the denoising model $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ was only trained on low-resolution images with noise schedule $\bar{\alpha}_t,\bar{\beta}_t$, which means the input signal-to-noise ratio that $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ is suited for at time $t$ is $\frac{\bar{\alpha}_t^2}{\bar{\beta}_t^2}$, but the signal-to-noise ratio of $\mathcal{D}[\boldsymbol{x}_t]$ is $\frac{s^2\bar{\alpha}_t^2}{\bar{\beta}_t^2}$. So directly using the model at time $t$ is not optimal.
What can we do about it? Simple: since the signal-to-noise ratio changes with time $t$, we can find another time $\tau$ whose signal-to-noise ratio equals $\frac{s^2\bar{\alpha}_t^2}{\bar{\beta}_t^2}$ — that is, we solve the equation
\begin{equation}\frac{\bar{\alpha}_{\tau}^2}{\bar{\beta}_{\tau}^2} = \frac{s^2\bar{\alpha}_t^2}{\bar{\beta}_t^2}\end{equation}
Once we solve for $\tau$, we know that the model at time $\tau$ is better suited to an input with signal-to-noise ratio $\frac{s^2\bar{\alpha}_t^2}{\bar{\beta}_t^2}$. So the denoising of $\mathcal{D}[\boldsymbol{x}_t]$ should use the model at time $\tau$ rather than time $t$. Additionally, $\mathcal{D}[\boldsymbol{x}_t]$ itself can be further refined: from equation $\eqref{eq:dx}$ we can see that when $s > 1$, the sum of squares of the two coefficients $\rho_t^2=\bar{\alpha}_t^2+\frac{\bar{\beta}_t^2}{s^2}$ is no longer 1, whereas during training the sum of squares of the coefficients is always 1. So we can divide by $\rho_t$ to bring it closer to the form seen during training. Ultimately, the main term of the denoising model constructed from $\mathcal{D}[\boldsymbol{x}_t]$ should be
\begin{equation}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\frac{\mathcal{D}[\boldsymbol{x}_t]}{\rho_t}, \tau\right)\label{eq:down-denoise}\end{equation}
Decomposition Approximation
Now we have two denoising models available: one is $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$, which directly uses the low-resolution model $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t^{\text{low}}, t)$ as a high-resolution model; the other is $\eqref{eq:down-denoise}$, derived in the previous section, which first downsamples and then denoises. Next, let's try to assemble them together.
Suppose we have a perfect denoising model $\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t)$ that has been trained on high-resolution images. We can then decompose it as
\begin{equation}\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t) = \underbrace{\color{red}{\mathcal{U}\left[\mathcal{D}\left[\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t)\right]\right]}}_{\text{low-resolution main term}} + \underbrace{\Big\{\color{green}{\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t) - \mathcal{U}\left[\mathcal{D}\left[\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t)\right]\right]}\Big\}}_{\text{high-resolution detail term}}\end{equation}
At first glance, this decomposition seems like a trivial identity, but it actually carries a very intuitive meaning: the first term takes the exact reconstruction result, downsamples it, and then upsamples it — in plain terms, shrinks it and then enlarges it. This is a lossy transformation, but the result is still enough to depict the overall outline of the subject, so it is the main term. The second term subtracts this overall outline from the exact result, and what remains clearly represents local detail.
Combining this with our earlier discussion, we believe that equation $\eqref{eq:down-denoise}$ from the previous section is a good approximation for the low-resolution main term, so we write
\begin{equation}\mathcal{D}\left[\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t)\right]\approx \frac{1}{s}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\frac{\mathcal{D}[\boldsymbol{x}_t]}{\rho_t}, \tau\right)\end{equation}
Note that we must not omit the leading factor $1/s$: this is because a denoising model typically predicts standard normal noise (i.e., $\boldsymbol{\varepsilon}$), so its output is itself approximately zero-mean and unit-variance. After downsampling $\mathcal{D}$, the variance becomes $1/s^2$, while the output of $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}$ is also unit-variance. So we need to divide by $s$ to bring the variance back to $1/s^2$, in order to improve the quality of the approximation.
For the high-resolution detail term, we write:
\begin{equation}\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t) - \mathcal{U}\left[\mathcal{D}\left[\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t)\right]\right]\approx \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \mathcal{U}\left[\mathcal{D}\left[\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\right]\right]\end{equation}
This is again based on the idea discussed earlier — directly using the low-resolution denoising model as a high-resolution model, which preserves texture detail reasonably well. So we take $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}$ to be a good approximation of $\boldsymbol{\epsilon}^{\text{high}}$ for high-resolution detail.
Combining these two approximations, we can write in full:
\begin{equation}\boldsymbol{\epsilon}^{\text{high}}(\boldsymbol{x}_t, t)\approx \frac{1}{s}\mathcal{U}\left[\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\frac{\mathcal{D}[\boldsymbol{x}_t]}{\rho_t}, \tau\right) \right]+ \Big\{\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) - \mathcal{U}\left[\mathcal{D}\left[\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\right]\right]\Big\}\triangleq \boldsymbol{\epsilon}_{\boldsymbol{\theta}}^{\text{approx}}(\boldsymbol{x}_t, t)\label{eq:high-key}\end{equation}
This is the key approximation for the high-resolution denoising model that we've been looking for!
In fact, using equation $\eqref{eq:high-key}$ directly to generate high-resolution images already produces quite good results, but we can also introduce a tunable hyperparameter to do even better. The idea is to mimic the technique described in Diffusion Models for Generation, Chatting Casually (Part 9): Controlling the Generated Result with Conditions, where an unconditional model is used to strengthen conditional generation. Here we treat $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}^{\text{approx}}(\boldsymbol{x}_t, t)$ as a conditional denoising model, where the guidance signal is the main term obtained by upsampling the low-resolution result (this is exactly what "Upsample Guidance," or UG, refers to in the paper's title), while $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ is treated as the unconditional denoising model. To strengthen the conditioning, we introduce a tunable parameter $w > 0$ and express the final denoising model used as
\begin{equation}\begin{aligned} \tilde{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) =&\, (1 + w)\, \boldsymbol{\epsilon}_{\boldsymbol{\theta}}^{\text{approx}}(\boldsymbol{x}_t, t) - w\,\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) \\ =&\, \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t) + (1 + w)\mathcal{U}\left[\frac{1}{s}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\frac{\mathcal{D}[\boldsymbol{x}_t]}{\rho_t}, \tau\right) - \mathcal{D}\left[\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)\right]\right] \end{aligned}\end{equation}
According to the experimental results in the original paper, values of $w=0.2$ near a certain value work best.
Extending to LDM
Although the formulation above seems, on the surface, agnostic to whether we're dealing with a pixel-space diffusion model or a latent-space diffusion model (LDM), in fact, from a theoretical standpoint, the derivation above only strictly applies to pixel-space diffusion models. LDM adds a nonlinear encoder, and the pooled features of a large image passed through the encoder need not equal the encoder features of the small image. So our assumption of building the main term for the high-resolution denoising model via downsample-then-upsample may no longer hold.
To see what changes in the LDM setting, we can look at two experimental results from the original paper. The first examines reconstructions obtained by upsampling/downsampling the encoder's features and then feeding them into the decoder, as shown in the figure below. The results show that whether upsampling or downsampling, performing such operations directly in feature space degrades the image, which suggests that the weight of the main term — constructed by downsampling, denoising, then upsampling — may need to be reduced somewhat.
Upsampling/downsampling the encoder's features leads to degraded decoder outputs
The second experiment directly uses the low-resolution LDM, unmodified, as a high-resolution model, and the resulting decoder output can be seen in the "w/o UG" portion of the figure below. As you can see, unlike the pixel-space diffusion model, and largely thanks to the decoder's robustness to features, in the LDM setting, using $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ directly as a high-resolution model works much better — semantics and sharpness are both reasonably well preserved, with only occasional "deformities" appearing.
Difference in results between generating large images with a small-image LDM with and without Upsample Guidance (UG)
Based on the conclusions from these two experiments, the original paper changes $w$ in the LDM setting to a function that depends on time $t$:
\begin{equation}w_t = \left\{\begin{aligned} w,\quad t \geq (1-\eta) T \\ -1,\quad t < (1-\eta) T \end{aligned}\right.\end{equation}
When $w = -1$, Upsample Guidance becomes effectively absent — this amounts to only applying Upsample Guidance during the early stage of the diffusion process. This allows the early stage to better prevent deformities via Upsample Guidance, while letting the later stage fully exploit $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ to produce sharper, crisper results, and additionally saves computation — truly a "three birds with one stone" solution.
Demonstrating Results
Finally, the experiments. Actually, the "w/ UG" portion of the figure in the previous section already demonstrated the effect of Upsample Guidance in the LDM setting: we can see that Upsample Guidance indeed corrects the deformities that arise from using $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ directly for high-resolution generation, while preserving semantic correctness and image sharpness.
As for results in pixel space, see the figure below:
Effect of Upsample Guidance on a pixel-space diffusion model, as demonstrated in the original paper
Because of Upsample Guidance, the overall method resembles first generating a low-resolution image and then applying super-resolution to obtain a high-definition image — except that it's done in an unsupervised manner. As a result, it can basically guarantee FID scores no worse than those of the low-resolution generation results:
Relationship between FID and hyperparameters (here wt and θ correspond to this post's w plus 1)
Finally, I also tried this out with my own previously trained 128×128 CelebA face diffusion model, which further confirmed the effectiveness of Upsample Guidance:
My own experimental results. Left: generation at the training resolution (128). Middle and right: generation results at 256 and 512 resolution respectively, using Upsample Guidance
In terms of quality, this certainly falls short of a high-resolution model trained directly at that resolution, but it's better than simply upsampling a low-resolution image. As for inference cost, compared to using $\boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\boldsymbol{x}_t, t)$ trained directly on high-resolution images for generation, Upsample Guidance adds an extra low-resolution computation, and the proportional increase in compute cost is roughly $1/s^2$. In the LDM case, since Upsample Guidance is not applied during the later stage of generation, this proportion is even smaller. All things considered, Upsample Guidance qualifies as a reasonably priced "free lunch" for large-image generation.
Reflections
Having gone through the whole Upsample Guidance framework, I wonder what your impression is. My own feeling is that it's very much in the style of a physicist — wildly imaginative, boldly hypothesizing, yet somehow grasping the essence of the matter without even realizing it. I think I could probably write up a decent explanation of this kind of work, but coming up with it myself, independently, would be utterly impossible, since at best I only have a rather rigid mathematical mindset.
A very natural question about Upsample Guidance is: why exactly does it work? Take my own CelebA face-generation model trained in pixel space as an example — it was only ever trained on small 128×128 images and has never seen a 256×256 large image at all. So why can it generate a 256×256 large image that so aptly matches our expectations? Note that this differs from the case of ImageNet, which is a multi-scale dataset — for instance, a 128×128 image might depict a fish, or it might depict a person holding a fish in their hand. That is, even though all the inputs are 128×128, the model has seen fish at different relative scales, allowing it to better adapt to different resolutions. CelebA, by contrast, is a single-scale dataset — every face is aligned in size, position, and orientation. Yet even so, Upsample Guidance still manages to successfully generalize it to higher resolutions.
I believe this has something to do with DIP (Deep Image Prior). Roughly speaking, DIP argues that the CNN architectures commonly used in computer vision have already been highly selected for and are deeply suited to vision itself — so much so that even a model that has never been trained on real data can accomplish certain vision tasks, such as denoising, inpainting, and even simple super-resolution. Upsample Guidance, which allows a diffusion model that has never seen large images to generate large images that are basically consistent with our expectations, seems to also benefit from this architectural prior inherent to CNNs. Put simply, as demonstrated in the experiment in the first section of this post, Upsample Guidance depends on the fact that directly using a low-resolution model as a high-resolution model still preserves some genuinely useful texture detail in the generated result — and this is not a trivial property.
To verify this, I specifically tried a purely Transformer-based diffusion model I had trained before (something like DiT + RoPE-2D), and found that it completely failed to reproduce the Upsample Guidance effect — suggesting that this phenomenon is, at least to some extent, dependent on the CNN-based U-Net architecture. That said, readers using Transformers needn't be discouraged: although they can't take the Upsample Guidance route, they can take the NLP route of length generalization. The paper FiT: Flexible Vision Transformer for Diffusion Model shows that by training a diffusion model with a Transformer + RoPE-2D combination, one can reuse length-generalization techniques such as NTK and YaRN to achieve high-resolution image generation with no training, or only minimal fine-tuning.
Summary
This post introduced a technique called Upsample Guidance, which allows a trained low-resolution diffusion model to directly generate high-resolution images without any additional fine-tuning cost. Experiments show that it can reliably boost the resolution by at least a factor of two. Although its results still fall somewhat short of a diffusion model trained directly at high resolution, this nearly free lunch is still well worth learning about. This post has reorganized the reasoning and derivation behind the method from my own perspective, and offered some reflections on why it works.
(Afterword: As originally planned, this post was supposed to be published two days ago. The reason for the two-day delay is that, while writing it, I discovered that many details I initially thought I understood were, in fact, still somewhat vague — so I spent an extra two days on derivation and experimentation to arrive at a more precise understanding. This goes to show that systematically and clearly restating something one is learning is itself a process of continuous self-refinement and improvement — which is, perhaps, the whole point of keeping up the habit of writing.)
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.