Transformer Upgrade Path: 17. Some Simple Thoughts on Multimodal Positional Encoding
In the second post of this series, Transformer Upgrade Path: 2. Rotary Position Embedding That Draws on the Best of Many Approaches, I proposed Rotary Position Embedding (RoPE) — a scheme that realizes relative positional encoding via absolute positions. RoPE was originally designed for one-dimensional sequences such as text and audio (RoPE-1D). Later, in Transformer Upgrade Path: 4. Rotary Position Embedding for Two-Dimensional Positions, we extended it to two-dimensional sequences (RoPE-2D), which is suitable for ViT-style image inputs. However, whether RoPE-1D or RoPE-2D, both share the property of being single-modality — that is, pure-text or pure-image input scenarios. So how should RoPE be adapted for multimodal scenarios such as mixed image-text input?
I did a bit of searching and found very little work discussing this issue. The mainstream approach seems to be flattening all the inputs and then applying RoPE-1D to the resulting one-dimensional sequence, so even RoPE-2D is rarely used. Setting aside whether this approach becomes a bottleneck as image resolution further increases, it ultimately feels less than elegant. So in what follows, we try to find a natural way of combining the two.
Rotation Position
The word "rotary" in the name RoPE comes from the rotation matrix $\boldsymbol{\mathcal{R}}_n=\begin{pmatrix}\cos n\theta & -\sin n\theta\\ \sin n\theta & \cos n\theta\end{pmatrix}$, which satisfies
\begin{equation}\boldsymbol{\mathcal{R}}_m^{\top}\boldsymbol{\mathcal{R}}_n=\boldsymbol{\mathcal{R}}_{n-m}\end{equation}more
This gives us, for the inner product of $\boldsymbol{q},\boldsymbol{k}$ (assumed to be column vectors):
\begin{equation}\left(\boldsymbol{\mathcal{R}}_m\boldsymbol{q}\right)^{\top} \left(\boldsymbol{\mathcal{R}}_n\boldsymbol{k}\right)= \boldsymbol{q}^{\top}\boldsymbol{\mathcal{R}}_m^{\top}\boldsymbol{\mathcal{R}}_n \boldsymbol{k}=\boldsymbol{q}^{\top}\boldsymbol{\mathcal{R}}_{n-m}\boldsymbol{k}\end{equation}
In the leftmost expression, $\boldsymbol{\mathcal{R}}_m\boldsymbol{q},\boldsymbol{\mathcal{R}}_n\boldsymbol{k}$ is applied independently, with no interaction between $m,n$ involved, so formally it is an absolute position; but the equivalent form on the far right depends only on the relative position $n-m$, so once combined with dot-product attention, it effectively behaves as a relative position. This property also gives RoPE translation invariance: since $(n+c) - (m+c) = n-m$, if we add a constant to all absolute positions before applying RoPE, the result of attention should in theory remain unchanged (in practice there may be tiny errors due to limited computational precision).
The above is the form for $\boldsymbol{q},\boldsymbol{k}\in\mathbb{R}^2$. For $\boldsymbol{q},\boldsymbol{k}\in \mathbb{R}^d$ (where $d$ is even), we need a $d\times d$ rotation matrix, and for this we introduce $d/2$ different values of $\theta$ and construct a block-diagonal matrix:
\begin{equation}\small{\boldsymbol{\mathcal{R}}_n^{(d\times d)} = \begin{pmatrix} \cos n\theta_0 & -\sin n\theta_0 & 0 & 0 & \cdots & 0 & 0 \\ \sin n\theta_0 & \cos n\theta_0 & 0 & 0 & \cdots & 0 & 0 \\ 0 & 0 & \cos n\theta_1 & -\sin n\theta_1 & \cdots & 0 & 0 \\ 0 & 0 & \sin n\theta_1 & \cos n\theta_1 & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & \cos n\theta_{d/2-1} & -\sin n\theta_{d/2-1} \\ 0 & 0 & 0 & 0 & \cdots & \sin n\theta_{d/2-1} & \cos n\theta_{d/2-1} \\ \end{pmatrix}}\end{equation}
In terms of implementation, this amounts to grouping $\boldsymbol{q},\boldsymbol{k}$ into pairs, each pair using a different $\theta$ for a two-dimensional rotation transform — this is all existing RoPE material, so I won't elaborate further. In principle, we only need to find the solution for the lowest dimension, and it can then be extended to arbitrary dimensions via the block-diagonal construction, so the analysis below only considers the minimal dimension.
Two-Dimensional Position
The concept of "dimension" can carry multiple meanings. For example, when we just said $\boldsymbol{q},\boldsymbol{k}\in \mathbb{R}^d$, we meant that $\boldsymbol{q},\boldsymbol{k}$ are $d$-dimensional vectors. But the RoPE-1D and RoPE-2D that this post focuses on don't refer to that dimension — they refer to the number of dimensions required to record a position.
Text and its position IDs
For example, to specify the position of a token in text, we only need a scalar $n$ recording that it's the $n$-th token. But for an image, even after patchification, it typically retains two directional dimensions — width and height — so we need a coordinate pair $(x,y)$ to accurately encode the position of a given patch:
Image and its position coordinates
The construction $\boldsymbol{\mathcal{R}}_n$ introduced in the previous section encodes only a single scalar $n$, so it is RoPE-1D. To handle image input more appropriately, we need to generalize to the corresponding RoPE-2D:
\begin{equation}\boldsymbol{\mathcal{R}}_{x,y}=\left( \begin{array}{cc:cc} \cos x\theta & -\sin x\theta & 0 & 0 \\ \sin x\theta & \cos x\theta & 0 & 0 \\ \hdashline 0 & 0 & \cos y\theta & -\sin y\theta \\ 0 & 0 & \sin y\theta & \cos y\theta \\ \end{array}\right) = \begin{pmatrix}\boldsymbol{\mathcal{R}}_x & 0 \\ 0 & \boldsymbol{\mathcal{R}}_y\end{pmatrix}\end{equation}
Clearly, this is simply $\boldsymbol{\mathcal{R}}_x$ and $\boldsymbol{\mathcal{R}}_y$ combined in block-diagonal form, so it can naturally be extended to 3D or even higher dimensions as well. In implementation terms, it's even simpler: we just split $\boldsymbol{q},\boldsymbol{k}$ into two halves (or three equal parts for 3D, four for 4D, and so on), each half being a vector of $\mathbb{R}^{d/2}$, then apply RoPE-1D with $x$ to one half and RoPE-1D with $y$ to the other half, and finally concatenate them.
It's worth noting that, for symmetry and simplicity, we used the same $\theta$ for both instances of $x,y$ in constructing $\boldsymbol{\mathcal{R}}_{x,y}$ above, but this is in principle not required — if appropriate, we can assign $x,y$ slightly different values of $\theta$.
Forcing a Dimensionality Reduction
We now see that a text position is a scalar $n$, while an image position is a vector $(x,y)$ — the two are inconsistent, so when handling mixed image-text input we need some technique to reconcile this discrepancy.
The most direct approach, as mentioned at the start of the article, is to simply flatten the image into a one-dimensional sequence of vectors and treat it just like ordinary text, applying whatever positional encoding scheme is used for text. This approach is naturally very general — it's not limited to RoPE, but works for any absolute positional encoding as well. As far as I can tell, some existing multimodal models such as Fuyu-8b, DeepSeek-VL, and Emu2 all take this approach, possibly with some differences in the details — for instance, when crossing rows of patches, one might insert a special token such as [SEP] to mark the separation:
Both text and image flattened into one dimension
This approach also fits well with the currently dominant decoder-only architecture, because decoder-only models are not permutation-invariant even without positional encoding, so we must manually specify what we consider the best input order — and once we've specified an input order, using a one-dimensional positional encoding aligned with that order is a natural choice. Moreover, for pure text input, a model using this scheme is no different from an ordinary text-only LLM, which allows us to continue training an already-trained text LLM into a multimodal model.
However, from my point of view, the concept of positional encoding itself shouldn't be tied to a particular use of attention — it should apply universally to decoders, encoders, and indeed any attention mask. On the other hand, preserving the two-dimensional nature of position maximizes the retention of our prior about which positions are "close" to each other. For instance, we might think that positions $(x+1,y)$ and $(x,y+1)$ should both be similarly close to $(x,y)$, but if we flatten (say, horizontally then vertically), $(x,y)$ becomes $xw + y$, while $(x+1,y)$ and $(x,y+1)$ become $xw+y+w$ and $xw+y+1$ respectively — the distance of the former from $xw + y$ now depends on $w$, while the latter is a fixed $1$. Of course, we could specify some other ordering, but no matter how we order things, we cannot fully preserve the proximity of all neighboring positions — after all, once we drop a dimension, much of the expressible similarity structure is lost.
Unifying by Raising the Dimension
From a vector-space point of view, a one-dimensional scalar can be viewed as a special case of a two-dimensional vector. So rather than flattening everything down to one dimension, if instead we unify all input positions up to two dimensions, we in principle have more room to work with.
To this end, we can consider a common layout convention: treat images as delimiters that split the text into segments, with each contiguous run of text treated as a single "line," and an image treated as spanning multiple "lines" of text. The whole mixed image-text input then behaves like a multi-line document, where every text token or image patch has its own line number $x$ and its position within that line $y$. This assigns a two-dimensional position $(x,y)$ to every input unit (token or patch), so we can uniformly encode position using RoPE-2D (in principle any 2D-style positional encoding would work), while also preserving the two-dimensional nature of image positions.
Constructing a unified two-dimensional position by simulating layout
Clearly, the main advantage of this scheme is that it's very intuitive — it directly corresponds to actual visual layout, making it easy to understand and generalize. But it also has an obvious drawback: for pure text input, it cannot degenerate into RoPE-1D, but instead becomes a RoPE-2D where $x$ is always 1, which makes it questionable whether one can feasibly start from an already-trained text LLM to train a multimodal LLM. In addition, when using images as split points, if there are many images, the text may end up being split into overly "fragmented" pieces — for example, the length of each text segment may vary wildly, or text that should have been continuous is forcibly broken into new lines. These issues could become bottlenecks limiting performance.
Merging the Two into One
If we want to losslessly preserve the positional information of image patches, then unifying everything into two dimensions and using RoPE-2D (or some other 2D-style positional encoding) seems to be the inevitable choice — so the scheme in the previous section is already headed in the right direction. What we need to think through further is how to make it degenerate into RoPE-1D for pure text input, so as to remain compatible with existing text LLMs.
First, recall that $\boldsymbol{\mathcal{R}}_{x,y}$ is a block-diagonal combination of $\boldsymbol{\mathcal{R}}_x$ and $\boldsymbol{\mathcal{R}}_y$, so $\boldsymbol{\mathcal{R}}_{n,n}$ is a block-diagonal combination of two instances of $\boldsymbol{\mathcal{R}}_n$; and the $\boldsymbol{\mathcal{R}}_n^{(d\times d)}$ of RoPE-1D is likewise a block-diagonal combination of multiple instances of $\boldsymbol{\mathcal{R}}_n$ with different values of $\theta$. From this we can see that as long as we assign $x,y$ different values of $\theta$ chosen from $\boldsymbol{\mathcal{R}}_n^{(d\times d)}$, then $\boldsymbol{\mathcal{R}}_{n,n}$ can be regarded as part of RoPE-1D (i.e. $\boldsymbol{\mathcal{R}}_n^{(d\times d)}$). This suggests that, in order for RoPE-2D to be able to degenerate into RoPE-1D, the position of text should take the form $(n,n)$, rather than assigning a line number in some other way as in the previous section.
Then, within an image, we use ordinary RoPE-2D. For a single image with $w\times h$ patches, its flattened two-dimensional position coordinates are
$$\begin{array}{c|cccc|cccc|c|cccc} \hline x & 1 & 1 & \cdots & 1 & 2 & 2 & \cdots & 2 & \quad \cdots \quad & h & h & \cdots & h \\ \hline y & 1 & 2 & \cdots & w & 1 & 2 & \cdots & w & \quad \cdots \quad & 1 & 2 & \cdots & w \\ \hline \end{array}$$
If this image comes right after a sentence of length $L$, and the position of the last token of that sentence is $(L,L)$, then it seems natural that the positions of this image, immediately following the sentence, should be
$$\begin{array}{c|cccc|c|cccc} \hline x & L+1 & L+1 & \cdots & L+1 & \quad \cdots \quad & L+h & L+h & \cdots & L+h \\ \hline y & L+1 & L+2 & \cdots & L+w & \quad \cdots \quad & L+1 & L+2 & \cdots & L+w \\ \hline \end{array}$$
But this isn't quite perfect: the position of the sentence's last token is $(L,L)$, and the position of the image's first patch is $(L+1,L+1)$, so they differ by $(1,1)$. Now suppose another sentence follows this image, and let the position of that sentence's first token be $(K,K)$, while the position of the image's last patch is $(L+h,L+w)$. When $w\neq h$, no matter how we set $K$, we can never make the difference between $(K,K)$ and $(L+h,L+w)$ equal to $(1,1)$ — that is, the image is asymmetric with respect to the sentences on its left and right, which feels inelegant.
To improve on this, we can multiply the image's $x,y$ by a positive number $s,t$:
$$\begin{array}{c|cccc|cccc|c|cccc} \hline x & s & s & \cdots & s & 2s & 2s & \cdots & 2s & \quad \cdots \quad & hs & hs & \cdots & hs \\ \hline y & t & 2t & \cdots & wt & t & 2t & \cdots & wt & \quad \cdots \quad & t & 2t & \cdots & wt \\ \hline \end{array}$$
As long as $s,t\neq 0$, this scaling is lossless with respect to positional information, so this kind of operation is permissible. Once we introduce this scale factor, assuming the position of the sentence's last token is still $(L,L)$, the image's positions become the same sequence shifted by adding $L$. In that case, the difference between "the position of the sentence's last token" and "the position of the image's first patch" is $(s,t)$. If we want the difference between "the position of the first token of the sentence following the image" and "the position of the image's last patch" to likewise be $(s,t)$, then we should have
\begin{equation}\begin{pmatrix}L + hs \\ L + wt \end{pmatrix} + \begin{pmatrix}s \\ t \end{pmatrix} = \begin{pmatrix}K \\ K \end{pmatrix}\quad \Rightarrow \quad (h+1)s = (w+1)t\end{equation}
Given the arbitrariness of $h,w$, and wanting to guarantee that all position IDs remain integers, the simplest solution is naturally $s=w+1,t=h+1$, so the position of the first token of the new sentence will be $K=L+(w+1)(h+1)$. A concrete example is shown in the figure below:
Two-dimensional positions supporting degeneration into RoPE-1D
Extended Thoughts
The position of the last token of the left sentence is $L$, and the position of the first token of the right sentence is $K=L+(w+1)(h+1)$. If the middle part were also a sentence, we could deduce that it would have $(w+1)(h+1)-1$ tokens — equivalently, if a $w\times h$ image is sandwiched between two sentences, it's equivalent, in terms of the relative position between those two sentences, to inserting a sentence of $(w+1)(h+1)-1$ tokens between them. This number looks somewhat unnatural, since $wh$ would seem to be the "perfect" answer, but unfortunately that's just the simplest solution guaranteeing all position IDs remain integers. If we allow non-integer position IDs, then we could instead stipulate that a $w\times h$ image is equivalent to $wh$ tokens, from which we can derive
\begin{equation}s = \frac{wh + 1}{h+1}, \quad t = \frac{wh + 1}{w+1}\end{equation}
Some readers might ask: if two images of different sizes are adjacent to each other, is there no longer such a symmetric scheme? This is actually not hard to handle either — we just need to insert special tokens before and after each image, such as [IMG] and [/IMG], and encode the positions of these special tokens as if they were ordinary text tokens. This directly avoids the case of two images being immediately adjacent (because, by convention, the patches of any single image must always be sandwiched between [IMG] and [/IMG], and since these two tokens are treated as text, it follows that every image is necessarily sandwiched between two pieces of text). Also, [SEP] hasn't been mentioned in the discussion above — feel free to introduce it if needed. In fact, [SEP] is only necessary when doing patch-by-patch autoregressive image generation; if the image is purely used as input, or if image generation is done via a diffusion model, then [SEP] is unnecessary.
With that, our derivation of extending RoPE to mixed image-text input is complete. If a name is needed, we could call the final scheme "RoPE-Tie" (RoPE for Text-Image). It has to be said that the final RoPE-Tie isn't especially elegant — it comes across as a bit over-engineered. In terms of practical effect, compared to simply flattening everything into one dimension and using RoPE-1D, switching to RoPE-Tie may not actually bring much improvement; it is, more than anything, a product of my own compulsiveness about details. So, for multimodal models that have already been scaled up to a certain size, there's probably no need to make any changes — but if you haven't started yet, or have only just begun, it might be worth giving RoPE-Tie a try.
Summary
This post discussed how to combine RoPE-1D and RoPE-2D to better handle mixed image-text input formats. The main idea is to support two-dimensional position indices for images via RoPE-2D, while imposing appropriate constraints so that it degenerates into ordinary RoPE-1D in the pure-text case.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.