"Reinventing the Wheel": Some Thoughts on Multimodal Approaches (I): Lossless Input
In this post I'll share some closed-door speculation—or perhaps just guesses—about multimodal model architecture.
Google's recent Gemini 1.5 and OpenAI's Sora have once again reignited a lot of people's enthusiasm for multimodal models, and the fragmentary technical reports have sparked heated speculation about the architectures behind them. However, this post isn't being published just to jump on that bandwagon; in fact, some of these thoughts have been brewing for a long time, and I've only recently managed to straighten them out somewhat—it just so happens that this coincides with the two releases. more
Let me state upfront: "reinventing the wheel" here isn't false modesty. My own practical experience with large models is genuinely "unremarkable," and my multimodal experience is basically "a blank slate." This post is truly just "subjective speculation" based on some past experience with text generation and image generation.
Background
Let's first simplify the problem. The multimodality discussed in this post mainly refers to a text-image dual modality, where both input and output can be text or images. Many readers' first instinct might be: isn't a multimodal model just about burning money on GPUs, throwing Transformers at everything, and letting "brute force produce miracles"?
It's not that simple. Let's start with text generation. In fact, text generation has, from beginning to end, only ever had one mainstream route: the language model, i.e., modeling the conditional probability $p(x_t|x_1,\cdots,x_{t-1})$. Whether it's the earliest n-gram language models or the later Seq2Seq and GPT, they're all approximations of this same conditional probability. In other words, people have always been very clear about "which direction to go to achieve text generation"; only the underlying models differ—LSTM, CNN, Attention, and even the recently revived linear RNNs. So text generation really can go "all in on Transformers" and produce miracles through brute force, because the direction is standard and clear.
However, for image generation, there's no such "standard direction." Among the image generation models discussed on this blog alone, there are VAE, GAN, Flow, Diffusion, and the more niche EBM, PixelRNN/PixelCNN, and so on. The distinctions between these methods don't arise from differences in effect caused by using RNN, CNN, or Attention—rather, their underlying modeling theories are fundamentally different. The root cause of this diversity of image generation methods is the inherent difficulty of probabilistic modeling for continuous variables.
For a sentence $l$ of length $(x_1,x_2,\cdots,x_l)$, each $x_t$ comes from a finite vocabulary, so $p(x_t|x_1,\cdots,x_{t-1})$ is essentially a classification task. Under the combination of "the universal approximation power of neural networks + softmax," in theory any classification task can be modeled exactly—this is the theoretical guarantee underlying text generation. However, we usually treat images as continuous-valued vectors, so for an image, $x_t$ is a real number. Even though we could apply the same conditional factorization, how would we model $p(x_t|x_1,\cdots,x_{t-1})$? Note that here $p(x_t|x_1,\cdots,x_{t-1})$ is a probability density, and the necessary conditions for a probability density are non-negativity and integrating to one:
\begin{equation}\int p(x_t|x_1,\cdots,x_{t-1}) dx_t = 1\end{equation}
Aside from the normal distribution, how many other functions can we write down whose integral is always 1? And the functions we can actually write down, like the normal distribution, are not expressive enough to fit an arbitrarily complex distribution. In plain terms, neural networks are universal function approximators, but not universal approximators of probability densities—this is the essential difficulty of generative modeling for continuous variables. The various schemes for image generation are, in essence, all different tricks for sidestepping direct modeling of the probability density (Flow being the exception). Discrete variables don't have this difficulty, because the constraint on discrete probabilities is that they sum to 1, which softmax handles just fine.
The Discretization Route
At this point some readers might wonder: could we discretize images and then plug them into the text generation framework? Indeed we can—this is one of the current mainstream approaches (quite possibly the only one).
In fact, images are inherently discrete already. A $n\times n$-sized RGB image is really just $3n^2$ integers ranging from 0 to 255—equivalent, in other words, to a sentence of length $3n^2$ with a vocab_size of 256. Taking this even further, computers are fundamentally discrete: everything they can represent is discrete, whether text, image, audio, or video. So directly feeding their raw discrete representations into the text generation framework is theoretically sound. Early works like PixelRNN and PixelCNN did exactly this, performing autoregressive generation directly in the pixel space of images. In our earlier post Born to Save: From Standard Attention to Sparse Attention, we also discussed OpenAI's Sparse Transformer, one of whose main experiments was pixel-level autoregressive image generation.
However, the biggest problem with operating directly in pixel space is that the sequence is too long, making generation too slow. In most practical applications, the image resolution needs to be at least 256 to be of any practical value (unless we're just generating small emoji-sized images). Even taking $n=256$, we get $3n^2\approx 20\text{ten thousand}$—meaning that to generate a single 256-resolution image, we'd need 200,000 steps of autoregressive decoding! Although long-context techniques have made substantial progress recently, this cost is still extravagant, and the generation time would be hard to accept.
A natural idea, then, is "compress first, generate later"—i.e., use a separate model to compress the sequence length, generate in the compressed space, and then reconstruct the image via a decoder. Compression is naturally handled by an autoencoder (AE), but since we want to reuse the text-generation modeling paradigm, we also need the compressed representation to remain discrete. This calls for VQ-VAE, and later VQ-GAN, where VQ can also be swapped out for the more recent FSQ. Much like a text tokenizer, VQ-VAE/GAN plays the role of an "image tokenizer": it preserves the discreteness of the encoded result while substantially shrinking the sequence length (e.g., if the resolution is reduced to $1/4$, we get $3n^2\to (n/4)^2$, a 48x reduction), and the image can be reconstructed via a corresponding decoder (detokenization). Many multimodal works have already been built on this "image tokenizer" idea, such as the recent LWM and AnyGPT.
Whether we're working in raw pixel space or in the compressed encoding space, both share a common feature: they are two-dimensional. In other words, while text has only one dimension—left to right—an image has two: left-right and up-down. This means that when doing autoregressive generation, we need to manually design a generation order, such as left-to-right-then-top-to-bottom, top-to-bottom-then-left-to-right, counterclockwise from the center outward, or sorted by distance to the top-left corner, and so on. Different generation orders can noticeably affect generation quality, which introduces an extra hyperparameter and feels inelegant precisely because it isn't fully end-to-end. To address this, we can use cross-attention to combine the two-dimensional features and output an encoding with only a single direction; related work can be found in Planting a SEED of Vision in Large Language Model.
Compression Loss
It might seem like the "image tokenizer" approach has "solved" multimodal generation? Not quite—the problems are just beginning.
The biggest issue with image tokenizers like VQ-VAE and VQ-GAN is that, in order to noticeably speed up generation and shorten the sequence length, they perform heavy compression of the encoding resolution (the mainstream choice is $256\times 256\to 32\times 32$ or even $256\times 256\to 16\times 16$), which causes severe loss of image information. To get an intuitive sense of this, we can look at the reconstruction results from the SEED paper:
SEED's reconstruction results
As you can see, although the reconstructed image is indeed quite sharp and largely preserves the overall semantics of the input image, the local details are completely different. This means it's impossible to use such an image tokenizer to complete arbitrary text-image mixed tasks (such as OCR).
We can drive the point home more clearly with a quick information-theoretic accounting of just how severe this information loss is. First, referring to the experimental results in Generating Long Sequences with Sparse Transformers, we know that the average entropy of ImageNet-64 is 3.44 bits/byte; since the model used back then wasn't very large, this number could in principle be pushed even lower with a bigger model, so let's just call it 3 bits/byte. Then the average total information entropy of a 64×64 ImageNet image is $64\times 64\times 3\times 3$ bits. Next, we know that for a vocabulary of size $V$, the average information entropy per token is $\log_2 V$ bits. If we want to compress the encoding length down to $L$ while achieving lossless compression, we need at least
\begin{equation}L\times \log_2 V \geq 64\times 64\times 3\times 3\end{equation}
If $L=1024=32\times 32$, then we need at least $V\geq 2^{36}\approx 7\times 10^{10}$; if $L=256=16\times 16$, we need at least $V\geq 2^{144}\approx 2\times 10^{43}$! Clearly, the codebook sizes of current image tokenizers fall nowhere near such absurd magnitudes, so the inevitable result is severe information loss!
A natural objection is: why does it have to be lossless? Indeed, even humans can't achieve lossless perception—arguably, human understanding of images loses even more information than an image tokenizer does. But the point is that the most basic requirement for a model is to align with human cognition. In other words, lossy compression is fine, as long as it's lossless for humans—much like discarding infrared and ultraviolet light is completely lossless for the human eye. However, "lossless for humans" is itself a fairly vague notion with no computable metric. VQ-VAE directly reconstructs images using an L2 distance, and blurriness is inevitable due to the resulting information loss; VQ-GAN adds a GAN loss to improve sharpness, but it can only roughly preserve global semantics and can't fully align with human standards. What's more, no one knows when some new image task requiring even finer detail might emerge, so from the perspective of general intelligence, lossless compression is the inevitable ultimate choice.
This shows that in a truly general multimodal model, the image component must be considerably harder than the text component, because images carry far more information than text. But actually, images created by humans themselves (like drawings) aren't much more complex than text (like writing); the truly complex images are photographs captured directly from nature. So at bottom, text is purely a human artifact, whereas images are artifacts of nature—and since humans are not as clever as nature, text is correspondingly less difficult than images. Truly general artificial intelligence is, after all, meant to head in the direction of comprehensively surpassing human capability.
Diffusion Models
Back to the main thread. Given the current state of image generation technology, if we insist on lossless compression, then either we go back to autoregression in pixel space—which, as discussed above, has unacceptably slow generation speed—or the only remaining option is to return to continuous space, treating the image as a continuous-valued vector. And within the constraint of lossless compression, the only two options left are flow models and diffusion models.
Flow models are invertible by design, and diffusion models can also be cast as invertible ODEs; both map a standard Gaussian distribution to the target distribution, which means they have a sufficient source of entropy. Discrete and continuous generation differ in this regard: the entropy source for discrete autoregressive generation is seqlen and vocab_size, and since vocab_size's contribution grows only logarithmically, seqlen does most of the work—but seqlen is equivalent to cost, so the discrete entropy source is expensive. The entropy source for transformation-based continuous generation, by contrast, is Gaussian noise, which in principle is inexhaustible, cheap, and parallelizable. However, to guarantee invertibility at every layer, Flow models require substantial architectural modifications, which likely places a noticeable ceiling on their achievable quality (there's no direct proof of this, but it's true that Flow models have never produced stunningly impressive generation results). So the only option left is the diffusion model.
Note that diffusion is only a choice for the image generation scheme. For image understanding, from a lossless perspective, any encoding scheme carries a risk of distortion, so the safest bet for input is clearly the raw image itself. The most reliable approach, then, would be to feed in the raw image directly as patches—similar to how Fuyu-8b handles things:
Diagram of the Fuyu-8b model
But Fuyu-8b only supports multimodal input; its output is still purely textual. How might we equip it with image generation capability as well? Given that training a diffusion model is essentially a denoising task, one possibly workable approach is:
A generation approach I imagine
During training, we feed in text plus a noised image. The training objective for the text part is next-token prediction as usual; the training objective for the image part is to predict the original image (or the noise). At prediction time, the text portion is still generated recursively token by token, until [IMG] is predicted, at which point a number of noise vectors are fed in parallel and sampled following the usual diffusion sampling procedure. Note that the image generation portion is parallel, so in principle it's actually better not to be decoder-only—because with a decoder-only setup you'd need to manually specify an ordering, and different orderings could noticeably affect quality. With current accelerated diffusion sampling techniques, image generation can basically be completed in about 10 steps, so the generation speed should be acceptable.
(Update, 2024.08.26: Meta's newly released Transfusion is essentially the same scheme as above, except that it adds an extra latent encoder step for images, and achieves quite good results. Slightly later, Show-o is also broadly similar, the difference being that it discretizes the diffusion process as well.)
Patch Input
A key piece of the approach above is the use of a patch-based diffusion model, so the most basic thing to verify is whether such a diffusion model design is even feasible (there had previously been claims that diffusion models are heavily dependent on the existing U-Net architecture). To this end, I ran some experiments of my own and also did a bit of a literature survey; below is a summary of my preliminary conclusions.
Based on what I could find, the earliest attempts to build diffusion models via the combination of "patch input + Transformer" were All are Worth Words: A ViT Backbone for Diffusion Models and Scalable Diffusion Models with Transformers. The two papers came out around the same time and take largely similar approaches. The former (U-ViT) mainly emphasizes the role of the "long skip connection" from U-Net, while the latter (DiT) emphasizes the necessity of injecting the diffusion time step $t$ and conditioning label $y$ into the model via adaLN. However, only U-ViT actually tried feeding raw image patches directly as input, and even then only at a resolution of 64×64. For 256×256 and 512×512 resolutions, both U-ViT and DiT perform diffusion in the dimensionality-reduced feature space produced by the LDM autoencoder. This is indeed the current mainstream approach, but as noted earlier, this degree of compression comes with severe information loss, and it's hard to call these truly general-purpose features.
Using raw image patches directly, instead of features from a pretrained encoder, has another benefit: it avoids isolating features from one another. For instance, when we need to feed in two images $I_1,I_2$ simultaneously, the typical approach based on encoder features is to feed $encoder(I_1),encoder(I_2)$ into the model—but the problem is that $encoder$ already applies a layer of intra-image semantic interaction, so once we feed in $encoder(I_1),encoder(I_2)$, the interaction between $I_1,I_2$ is missing that layer. This is the problem of feature isolation between images; more details can be found in Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion. So it seems better to feed both text and images in their raw form and let the multimodal model itself decide all the interactions—this way, this gap simply doesn't exist.
Of course, the fact that feeding raw image patches directly hasn't become mainstream must reflect some underlying difficulty, so I ran some experiments of my own. The experimental task was diffusion generation on CelebA-HQ at resolutions of 64×64 and 128×128, reshaped into 16×16×48 and 16×16×192 respectively before being projected into the model. The model itself is an ordinary Pre-Norm Transformer, without a long skip connection; the backbone is GAU rather than MHA; the positional encoding is 2D-RoPE, and the time embedding is added directly to the patch input. The code is available here:
Link: https://github.com/bojone/Keras-DDPM/blob/main/ddpm-gau.py
My experimental results show that, for both 64×64 and 128×128 resolutions, this setup does converge properly, eventually producing generation quality comparable to a plain U-Net (I didn't compute FID, this is purely by eye), but it needs many more training steps to converge. For example, on the same single A800 GPU, a plain U-Net at 128×128 resolution produces roughly presentable results after about 1–2 days of training, whereas the Transformer-based architecture took over 10 days to reach anything barely presentable. The likely reason is the absence of CNN's inductive bias—the model needs many more training steps to learn to adapt to the priors of images. But for a large multimodal model, this probably isn't much of an issue, since LLMs already require plenty of training steps as a matter of course.
Summary
This post has presented my own idea for multimodal model design: feed in raw image patches directly as image input, keep the text portion doing conventional next-token prediction, and have the image portion reconstruct the original image from a noised input—a combination that, in theory, achieves multimodal generation in the most faithful way possible. From my preliminary results, it appears feasible to train a successful image diffusion model using a Transformer that takes raw image patches directly as input, which means this design combining diffusion with text generation also has a real chance of succeeding. Of course, these are just some rather rough thoughts of mine on the multimodal route, most of which haven't been validated in practice—please read with appropriate skepticism.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.