"Building Cars Behind Closed Doors" — Some Thoughts on Multimodality (Part 3): Positional Encoding
In earlier posts, we made the point that the main difference between multimodal LLMs and text-only LLMs is that the former haven't even converged on a commonly accepted methodology. This "methodology" covers not only the generation and training strategies we discussed before, but also some fundamental architectural design choices — such as the "multimodal positional encoding" that this post is about.
We already touched on this topic once before, in Transformer Upgrade Path: 17. Simple Thoughts on Multimodal Positional Encoding, where we proposed a scheme (RoPE-Tie). At the time, though, my thinking on this problem was still at an early stage, with some details left unconsidered and my understanding not fully mature. Looking back now, that earlier proposal is noticeably far from a perfect answer.
So in this post we'll go through the problem again from the top, and arrive at a result that I now consider more satisfactory.
Multimodal Positions
It might surprise many readers that multimodal models haven't even reached a consensus on positional encoding — but that really is the case. For text LLMs, the current mainstream positional encoding is RoPE (we won't re-introduce RoPE here, assuming readers are already familiar with it) — or more precisely, RoPE-1D, since the original design only applies to 1D sequences. Later we derived RoPE-2D, which can be used for 2D sequences such as images, and following the same logic as RoPE-2D, we can extend it in parallel to RoPE-3D for 3D sequences such as video. more
However, everything said above concerns a single modality at a time. Once we mix multiple modalities as input, difficulties arise: text is a 1D sequence, so its position is just a scalar $n$; an image is 2D ("width" and "height"), so its position requires a two-dimensional vector $(x,y)$; video adds a time dimension (or "frame" dimension) on top of images, so its position is a three-dimensional vector $(x,y,z)$. If we want a single model to handle data from all three modalities, we need some way to blend these three different forms of positional information.
As we all know, RoPE is implemented as an absolute positional encoding, but when combined with dot-product-based attention, positions effectively get subtracted during the inner product, giving the effect of relative positional encoding. But while vectors of the same size can be subtracted, how do you subtract vectors of different sizes? This is exactly where the difficulty of multimodal positional encoding lies.
Many works choose to "dodge" this difficulty by simply flattening all modalities and then using RoPE-1D. That's certainly one way to solve it, but it's ultimately not very elegant. Moreover, forcibly flattening may also lower the ceiling on model performance, since works such as VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks have already shown that introducing RoPE-2D helps improve model performance, especially for variable-resolution inputs.
Backward Compatibility
So, what we want is to design a multimodal positional encoding that can be used across mixed modalities, while still reducing to the corresponding RoPE-1D/2D/3D under a single modality — so as to fully unlock the capability of each modality.
As mentioned, the main difficulty with multimodal positional encoding is that position vectors of different sizes can't be subtracted. To preserve full positional information while still allowing subtraction, the only option is to unify everything by raising it to the highest dimension involved. Let's take text-image mixed modality as an example: since images are 2D, we also lift the text's positional encoding to two dimensions, and use RoPE-2D uniformly throughout. But can we lift dimensions any way we like? Not quite — we want it to have backward compatibility, meaning that when the input is pure text, it should be exactly equivalent to RoPE-1D.
To this end, let's compare RoPE-1D and RoPE-2D:
$$\scriptsize{\begin{array}{c}\begin{array}{c}\text{RoPE-1D}\\ (\boldsymbol{\mathcal{R}}_n)\end{array}= \begin{pmatrix} \cos \bbox[yellow]{n}\theta_0 & -\sin \bbox[yellow]{n}\theta_0 & 0 & 0 & \cdots & 0 & 0 & 0 & 0 \\ \sin \bbox[yellow]{n}\theta_0 & \cos \bbox[yellow]{n}\theta_0 & 0 & 0 & \cdots & 0 & 0 & 0 & 0 \\ 0 & 0 & \cos \bbox[yellow]{n}\theta_1 & -\sin \bbox[yellow]{n}\theta_1 & \cdots & 0 & 0 & 0 & 0 \\ 0 & 0 & \sin \bbox[yellow]{n}\theta_1 & \cos \bbox[yellow]{n}\theta_1 & \cdots & 0 & 0 & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & \cos \bbox[yellow]{n}\theta_{d/2-2} & -\sin \bbox[yellow]{n}\theta_{d/2-2} & 0 & 0 \\ 0 & 0 & 0 & 0 & \cdots & \sin \bbox[yellow]{n}\theta_{d/2-2} & \cos \bbox[yellow]{n}\theta_{d/2-2} & 0 & 0 \\ 0 & 0 & 0 & 0 & \cdots & 0 & 0 & \cos \bbox[yellow]{n}\theta_{d/2-1} & -\sin \bbox[yellow]{n}\theta_{d/2-1} \\ 0 & 0 & 0 & 0 & \cdots & 0 & 0 & \sin \bbox[yellow]{n}\theta_{d/2-1} & \cos \bbox[yellow]{n}\theta_{d/2-1} \\ \end{pmatrix} \\[16pt] \begin{array}{c}\text{RoPE-2D}\\ (\boldsymbol{\mathcal{R}}_{x,y})\end{array}= \begin{pmatrix} \cos \bbox[yellow]{x}\theta_0 & -\sin \bbox[yellow]{x}\theta_0 & 0 & 0 & \cdots & 0 & 0 & 0 & 0 \\ \sin \bbox[yellow]{x}\theta_0 & \cos \bbox[yellow]{x}\theta_0 & 0 & 0 & \cdots & 0 & 0 & 0 & 0 \\ 0 & 0 & \cos \bbox[yellow]{y}\theta_1 & -\sin \bbox[yellow]{y}\theta_1 & \cdots & 0 & 0 & 0 & 0 \\ 0 & 0 & \sin \bbox[yellow]{y}\theta_1 & \cos \bbox[yellow]{y}\theta_1 & \cdots & 0 & 0 & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & \cos \bbox[yellow]{x}\theta_{d/2-2} & -\sin \bbox[yellow]{x}\theta_{d/2-2} & 0 & 0 \\ 0 & 0 & 0 & 0 & \cdots & \sin \bbox[yellow]{x}\theta_{d/2-2} & \cos \bbox[yellow]{x}\theta_{d/2-2} & 0 & 0 \\ 0 & 0 & 0 & 0 & \cdots & 0 & 0 & \cos \bbox[yellow]{y}\theta_{d/2-1} & -\sin \bbox[yellow]{y}\theta_{d/2-1} \\ 0 & 0 & 0 & 0 & \cdots & 0 & 0 & \sin \bbox[yellow]{y}\theta_{d/2-1} & \cos \bbox[yellow]{y}\theta_{d/2-1} \\ \end{pmatrix}\end{array}}$$
Notice any common pattern? Just looking at this form, we can see that $\boldsymbol{\mathcal{R}}_n=\boldsymbol{\mathcal{R}}_{n,n}$ holds, i.e., RoPE-1D at position $n$ is actually equivalent to RoPE-2D at position $(n,n)$. So, in order to use RoPE-2D uniformly across text-image mixtures while still reducing to RoPE-1D for pure text, we need to set the position coordinates of the text portion to the form $(n,n)$.
Of course, in practice there's a slight subtlety: for RoPE-1D we have $\theta_i = b^{-2i/d}$, meaning $\theta_{2j}$ and $\theta_{2j+1}$ differ; but for RoPE-2D, to ensure the symmetry of $x,y$, the usual choice is to require $\theta_{2j}=\theta_{2j+1}$, which creates a contradiction. Here we have two options: either give up the symmetry of $x,y$ in RoPE-2D and still take $\theta_i = b^{-2i/d}$, or take $\theta_{2j}=\theta_{2j+1}=b^{-4j/d}$, in which case the positional encoding of the pure-text portion differs slightly from the existing RoPE-1D. For $\theta_i = b^{-2i/d}$, the difference between $\theta_i$ and $\theta_{i+1}$ is small, so the two options are essentially comparable, and which one to choose is a matter of personal taste — I personally lean toward the first.
Equivalence and Symmetry
Through the analysis above, we've settled on a scheme where text-image mixed modality uses RoPE-2D uniformly, and backward compatibility tells us that the 2D position of a text token at position $n$ should be $(n,n)$, which completes the positional encoding design for the text part. Next, we need to work out the positional encoding for the image part.
If the input contains only a single image with $w\times h$ patches, then its position coordinates are naturally just the coordinates of each patch itself, i.e.,
\begin{equation}\left[\begin{matrix} (1,1) & (1,2) & \cdots & (1, w) \\ (2,1) & (2,2) & \cdots & (2, w) \\ \vdots & \vdots & \ddots & \vdots \\ (h,1) & (h,2) & \cdots & (h, w) \\ \end{matrix}\right]\label{eq:rope2d}\end{equation}
Here we're showing absolute positions, but what actually matters is relative position, and relative position doesn't depend on positional offset — so we can add an offset $(\beta_1,\beta_2)$ to every coordinate without changing the effect. Also, we can scale every coordinate by $(\gamma_1,\gamma_2)$, which lets us adjust the spacing between adjacent positions as needed. Putting these two together, we get a generalized 2D position for images:
\begin{equation}\left[\begin{matrix} (\beta_1 + \gamma_1,\beta_2 + \gamma_2) & (\beta_1 + \gamma_1,\beta_2 + 2\gamma_2) & \cdots & (\beta_1 + \gamma_1,\beta_2 + w\gamma_2) \\[8pt] (\beta_1 + 2\gamma_1,\beta_2 + \gamma_2) & (\beta_1 + 2\gamma_1,\beta_2 + 2\gamma_2) & \cdots & (\beta_1 + 2\gamma_1,\beta_2 + w\gamma_2) \\[8pt] \vdots & \vdots & \ddots & \vdots \\[8pt] (\beta_1 + h\gamma_1,\beta_2 + \gamma_2) & (\beta_1 + h\gamma_1,\beta_2 + 2\gamma_2) & \cdots & (\beta_1 + h\gamma_1,\beta_2 + w\gamma_2) \end{matrix}\right]\end{equation}
Now let's consider the case where two spans of text sandwich a single image in between, and figure out how to choose $\beta_1,\beta_2,\gamma_1,\gamma_2$.
First, we assume a certain equivalence between text tokens and patches: after a reasonable patchify step, each patch has the same status as a token (An Image is Worth xxx Tokens). This means that, from the point of view of the two text spans, they're effectively sandwiching a sentence of $wh$ tokens. So if the last token of the left-hand text span is at position $(L,L)$, then the first token of the right-hand text span should be at position $(L+wh+1, L + wh + 1)$.
Next, we also need to bring in symmetry — specifically, the first patch of the image is at position $(\beta_1 + \gamma_1,\beta_2 + \gamma_2)$ and the last patch at position $(\beta_1 + h\gamma_1,\beta_2 + w\gamma_2)$. We require that the positional gap between the [first patch of the image] and the [last token of the left text span] equals the positional gap between the [first token of the right text span] and the [last patch of the image], i.e.,
\begin{equation}\begin{pmatrix}\beta_1 + \gamma_1 \\ \beta_2 + \gamma_2\end{pmatrix} - \begin{pmatrix}L \\ L\end{pmatrix} = \begin{pmatrix}L+wh+1 \\ L+wh+1\end{pmatrix} - \begin{pmatrix}\beta_1 + h\gamma_1 \\ \beta_2 + w\gamma_2\end{pmatrix}\label{eq:beta-gamma}\end{equation}
There are four unknowns $\beta_1,\beta_2,\gamma_1,\gamma_2$ here, but only two equations, so there are infinitely many solutions. We can simply take $\gamma_1=\gamma_2=1$, and then solve to get
\begin{equation}\beta_1 = L + \frac{1}{2}(wh - h),\quad \beta_2 = L + \frac{1}{2}(wh - w)\end{equation}
Let's call this scheme, for now, RoPE-Tie-v2, or RoPE-TV (RoPE for Text and Vision).
Pros and Cons
Based on this result, when a sentence is followed by an image of size $w\times h$, all we need is to compute $(\beta_1,\beta_2)$ via the formula above and add it to the usual 2D RoPE $\eqref{eq:rope2d}$, and we get the positional coordinates for the image part, as shown in the diagram below.
Diagram of the new RoPE-TV (RoPE-Tie-v2)
As a comparison, here is the diagram of the earlier RoPE-Tie scheme proposed in Transformer Upgrade Path: 17. Simple Thoughts on Multimodal Positional Encoding:
Diagram of the old RoPE-Tie
In fact, the original RoPE-Tie was also motivated by compatibility and symmetry, but it didn't strictly follow equivalence, and it implicitly assumed $\beta_1=\beta_2=L$, without requiring that $w\times h$ patches be equivalent to $wh$ tokens. In the end it arrived at a set of integer solutions (integer solutions aren't actually required to satisfy equivalence):
\begin{equation}\gamma_1 = w+1,\quad\gamma_2=h+1\end{equation}
Looking back on it now, the default settings in RoPE-Tie weren't really ideal. So in this post we re-chose $\gamma_1=\gamma_2=1$, made sure equivalence held, and then derived $\beta_1,\beta_2$ from that.
So what are the advantages of the new scheme? First, in RoPE-Tie, the relative positions within an image depend on the image's size, whereas in the new scheme the patch spacing is fixed at $(0,1)$ and $(1,0)$, which makes patch scale more consistent. For example, consider a 128×128 image and the top half of that same image (a 128×64 sub-image): since the two have different heights, under RoPE-Tie their horizontal position spacing would differ, meaning that two patches at the same position with the same semantic meaning would end up at inconsistent distances (scales) after applying RoPE-Tie — which seems unreasonable. The new scheme doesn't have this issue.
Second, in RoPE-Tie the gap between the image and the surrounding text is the same $(\gamma_1,\gamma_2)$ as the gap between patches inside the image, whereas in the new scheme there's a relatively large gap $\frac{1}{2}(wh - h, wh-w)$ between text and image (and vice versa), while within text and within the image the spacing stays fixed and uniform. Intuitively, this kind of larger positional jump between different modalities can better achieve "modality isolation," letting a single model handle within-modality content well while still preserving cross-modal interaction — which has much the same spirit as the common practice of marking images with special tokens like [IMG] and [/IMG] on either side.
The Three-Dimensional Predicament
The RoPE-Tie post didn't discuss positional encoding for "text-video" mixed modality; let's fill that gap here.
Intuitively, there are two ways to handle video input. The first is simply to treat the video as a sequence of images (adding [VIDEO]/[/VIDEO] markers if needed), so that we don't need a new positional encoding scheme specifically for video — we can just reuse the "text-image" mixed positional encoding result. But this loses the alignment relationship between different frames of the same video, which may not be entirely satisfactory. For example, "patch 1 of frame 1" should have roughly similar proximity to both "patch 1 of frame 2" and "patch 2 of frame 1" — but flattening the video into a sequence of images fails to reflect this.
The second way is to extend the "text-image" result in parallel to "text-video." For a video of size $w\times h\times t$ (frame size $w\times h$, with $t$ frames total), its position coordinates are three-dimensional: $(x,y,z)$. Following the same compatibility, equivalence, and symmetry principles, we can generalize equation $\eqref{eq:beta-gamma}$ to
\begin{equation}\begin{pmatrix}\beta_1 + \gamma_1 \\ \beta_2 + \gamma_2 \\ \beta_3 + \gamma_3\end{pmatrix} - \begin{pmatrix}L \\ L \\ L\end{pmatrix} = \begin{pmatrix}L+wht+1 \\ L+wht+1 \\ L+wht+1\end{pmatrix} - \begin{pmatrix}\beta_1 + h\gamma_1 \\ \beta_2 + w\gamma_2 \\ \beta_3 + t\gamma_3\end{pmatrix}\end{equation}
If we again set $\gamma_1=\gamma_2=\gamma_3=1$, we get
\begin{equation}\beta_1 = L + \frac{1}{2}(wht - h),\quad \beta_2 = L + \frac{1}{2}(wht - w),\quad \beta_3 = L + \frac{1}{2}(wht - t)\end{equation}
This fully preserves the three-dimensional nature of video positions, and looks more elegant — but I still think it leaves something to be desired.
This shortcoming comes from my own understanding of the time dimension of video: the "three dimensions" of a video are really "2 spatial dimensions + 1 temporal dimension," which is not the same as the "3 spatial dimensions" of a genuine 3D object in the real world. In my view, the time dimension of video is not on equal footing with the two spatial dimensions; the time dimension is more like the left-to-right writing direction of text. So my mental image of a perfect multimodal LLM is one that, just like a text LLM continuing a piece of text, should in principle be able to autoregressively continue generating video indefinitely, until an [EOS] token appears.
We mentioned two schemes for "text-video" mixed encoding above. The first, treating video simply as a sequence of images, allows unbounded autoregressive video generation. But the second, seemingly more elegant scheme, does not, because its $\beta_1,\beta_2,\beta_3$ depends on $t$ — meaning we'd need to know the number of frames to be generated ahead of time. In other words, the second scheme isn't strictly incompatible with autoregressive video generation, but it does require the frame count to be fixed in advance, which in my view doesn't match the ideal property of the time dimension (time should be able to advance without bound).
Some readers might wonder: why doesn't image generation mind that $\beta_1,\beta_2$ depends on $w,h$? That is, why doesn't image generation mind knowing the image size ahead of time? This is because an image has two spatial directions, and even if we generate an image autoregressively, we must at least know the size along one direction so as to tell the model when to "wrap to a new line," in order to produce a complete 2D image. Since the two spatial dimensions of an image are on equal footing, knowing just one is nearly as informative as knowing both, so it's acceptable to fix the image size in advance.
In addition, we could also build a "text-image" model using the "AR+Diffusion" approach described in Building Cars Behind Closed Doors — Some Thoughts on Multimodality (Part 1): Lossless Input. In that case, since the image-generation part is a diffusion model, the target image size must be known in advance anyway.
Related Work
A while back, Alibaba open-sourced a multimodal model called "Qwen2-VL," and its introduction mentioned a proposed multimodal rotary positional encoding (M-RoPE), which caught my interest. After reading the source code (link), I found that M-RoPE essentially follows the compatibility idea from RoPE-Tie, but doesn't preserve symmetry or equivalence.
Source code comments for M-RoPE
In the notation of this post, M-RoPE essentially takes $\beta_1=\beta_2=\beta_3=L,\gamma_1=\gamma_2=\gamma_3$ (for "text-video" mixed modality), and the position of the first token of the text following the video is simply set to the video's maximum position coordinate plus 1. This way, if you still want to generate video autoregressively, you indeed don't need to fix the frame count in advance — but at the cost of symmetry and equivalence.
How important are symmetry and equivalence, really? I honestly don't know — that requires thorough experiments to verify. But just as a piece of speculation, I suspect this might affect performance in extreme cases. For instance, with M-RoPE, if a video has a very small frame size but is very long in time, its spatial position coordinates would appear continuous relative to the preceding text span but would jump discontinuously relative to the following text span — which intuitively could make text-visual interaction less friendly.
Or take another example: a video of size $w=h=t=n$ is intuitively equivalent to $n^3$ tokens. But under the M-RoPE convention, if two spans of text sandwich such a video, it's treated as equivalent to sandwiching a text span of only $n$ tokens. In other words, within a relative distance of $n$, we're packing in $n^3$ tokens — could this excessive information density make the video harder for the model to understand?
Of course, for decoder-only LLMs where even NoPE can apparently work fine, maybe I'm overthinking all of this.
Summary
This post shares my follow-up thoughts on multimodal positional encoding. I proposed three guiding principles for constructing multimodal positional encodings — compatibility, equivalence, and symmetry — improved upon the previously proposed RoPE-Tie, and finally discussed the design and difficulties of positional encoding for "text-video" mixed modality, as well as the connection between Qwen2-VL's M-RoPE and RoPE-Tie.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.