Transformer Upgrade Path: 4. Two-Dimensional Rotary Position Embedding
In the earlier post Transformer Upgrade Path: 2. Rotary Position Embedding, Drawing on the Best of Many Approaches we proposed Rotary Position Embedding (RoPE) and the corresponding Transformer model, RoFormer. Since my main field of research is NLP, as far as I was concerned that was basically the end of the story. But recently Transformer models have also taken off in the vision domain, with all kinds of Vision Transformers (ViT) popping up one after another. This raises a natural question: what should the two-dimensional version of RoPE look like?
At first glance this seems like it should just be a simple generalization of the one-dimensional case, but the derivation and the understanding involved turn out to be far more intricate than one might expect. This post works through that analysis, which in turn deepens our understanding of RoPE itself.
Two-Dimensional RoPE
What exactly is a two-dimensional position? What does the corresponding two-dimensional RoPE look like? And where does the difficulty lie? In this section we'll briefly introduce two-dimensional positions, then directly present the result and the line of reasoning for two-dimensional RoPE. In the following sections we'll go through the derivation in detail. more
Two-Dimensional Position
In NLP, positional information for language is one-dimensional — in other words, we need to tell the model which word in the sentence a given token is. In CV, however, positional information for an image is two-dimensional: we need to tell the model which row and which column a given feature is located at. Here "two-dimensional" means that fully describing the position requires two numbers; it does not refer to the dimensionality of the position vector.
Some readers might wonder: can't we just flatten it and treat it as one-dimensional? Not really. Consider, for example, a $h\times h$ feature map: position $(x,y)$ becomes $xh + y$ after flattening, while positions $(x+1,y)$ and $(x,y+1)$ become $xh+y+h$ and $xh+y+1$ respectively, and their differences from $xh + y$ are $h$ and $1$. But intuitively, $(x+1,y)$ and $(x,y+1)$ should be equally distant from $(x,y)$ — yet after flattening we get the unequal values $h$ and $1$, which is clearly unreasonable.
So we need a position encoding specifically designed for the two-dimensional case; we can't simply flatten it into one dimension.
The Answer
After some derivation, which we present below, one solution for two-dimensional RoPE is:
\begin{equation}\boldsymbol{\mathcal{R}}_{x,y}=\left( \begin{array}{cc:cc} \cos x\theta & -\sin x\theta & 0 & 0 \\ \sin x\theta & \cos x\theta & 0 & 0 \\ \hdashline 0 & 0 & \cos y\theta & -\sin y\theta \\ 0 & 0 & \sin y\theta & \cos y\theta \\ \end{array}\right)\label{eq:rope-2d}\end{equation}
This solution is easy to understand: it's a block matrix made up of two one-dimensional RoPE matrices. In practice, this means splitting the input vector into two halves, applying a one-dimensional RoPE with position $x$ to one half, and a one-dimensional RoPE with position $y$ to the other. From this form it's also easy to generalize by analogy to three-dimensional, four-dimensional, and higher-dimensional positions.
The matrix $\eqref{eq:rope-2d}$ is an orthogonal matrix satisfying two key properties:
1. Relativity: that is, $\boldsymbol{\mathcal{R}}_{x_1,y_1}^{\top}\boldsymbol{\mathcal{R}}_{x_2,y_2}=\boldsymbol{\mathcal{R}}_{x_2-x_1,y_2-y_1}$. It is precisely because of this property that RoPE is able to express relative position through absolute position.
2. Invertibility: given $\boldsymbol{\mathcal{R}}_{x,y}$ we can solve back for $x,y$, meaning the encoding of positional information is lossless.
In a sense, equation $\eqref{eq:rope-2d}$ is the simplest solution satisfying the above two properties — that is, while there exist other, slightly different solutions satisfying these two properties, they are all more complex both in form and in implementation.
Line of Reasoning
Looking back, what RoPE really did was find a matrix $\boldsymbol{\mathcal{R}}_n=\begin{pmatrix}\cos n\theta & -\sin n\theta\\ \sin n\theta & \cos n\theta\end{pmatrix}$ satisfying the "relativity" condition:
\begin{equation}\boldsymbol{\mathcal{R}}_m^{\top}\boldsymbol{\mathcal{R}}_n=\boldsymbol{\mathcal{R}}_{n-m}\label{eq:re}\end{equation}
So it's natural to expect that the basic requirement for two-dimensional RoPE is likewise to satisfy relativity, i.e., to find a matrix $\boldsymbol{\mathcal{R}}_{x,y}$ satisfying the two-dimensional relativity condition $\boldsymbol{\mathcal{R}}_{x_1,y_1}^{\top}\boldsymbol{\mathcal{R}}_{x_2,y_2}=\boldsymbol{\mathcal{R}}_{x_2-x_1,y_2-y_1}$. However, if this were the only requirement, there would be many feasible solutions — for example, simply letting
\begin{equation}\boldsymbol{\mathcal{R}}_{x,y} = \begin{pmatrix}\cos (x+y)\theta & -\sin (x+y)\theta\\ \sin (x+y)\theta & \cos (x+y)\theta\end{pmatrix}\end{equation}
The problem with this solution is that we cannot invert $(x,y)$ from $x+y$, meaning this choice loses positional information. So we need one more requirement, "invertibility," to guarantee that the original position signal can be losslessly reconstructed from the position matrix.
For this, there are two fairly natural avenues to consider: 1. quaternions; 2. matrix exponentials. We'll introduce each of these in turn.
Quaternions
In deriving one-dimensional RoPE we mainly used complex numbers as our tool, and the quaternion is a generalization of the complex number that retains many of its properties, so using quaternions to derive two-dimensional RoPE is a fairly natural idea. Unfortunately, this turns out to be a dead end — but I'll still lay out the thought process here for reference.
Complex Numbers and Matrices
Back in high school we learned that a complex number $a+b\boldsymbol{i}$ corresponds one-to-one with a two-dimensional vector $(a,b)$ (to keep things consistent with the quaternion notation later, I've also bolded the imaginary unit $\boldsymbol{i}$ here). But this correspondence only preserves addition and subtraction (since vectors have no general-purpose multiplication operation). A more elegant correspondence is mapping complex numbers to matrices:
\begin{equation}a+b\boldsymbol{i} \quad \leftrightarrow \quad \begin{pmatrix} a & -b \\ b & a \end{pmatrix}\end{equation}
Under this mapping, addition, subtraction, multiplication, and division of complex numbers correspond exactly to those operations on matrices, e.g.,
\begin{equation}\begin{array}{ccc} (a+b\boldsymbol{i})(c+d\boldsymbol{i}) &=& (ac - bd) + (ad + bc)\boldsymbol{i} \\[5pt] \begin{pmatrix} a & -b \\ b & a \end{pmatrix}\begin{pmatrix} c & -d \\ d & c \end{pmatrix} &=& \begin{pmatrix} ac - bd & - ad - bc \\ ad + bc & ac - bd \end{pmatrix} \end{array}\end{equation}
So the matrix mapping is a complete isomorphism of the complex field, whereas the vector mapping is merely a convenient geometric picture.
The matrix representation of complex numbers is also an important foundation of RoPE. In Transformer Upgrade Path: 2. Rotary Position Embedding, Drawing on the Best of Many Approaches we already derived that the complex-number form of RoPE is $\boldsymbol{q}e^{n\boldsymbol{i}\theta}=(\cos n\theta + \boldsymbol{i}\sin n\theta)\boldsymbol{q}$, so by the matrix mapping of complex numbers, $\cos n\theta + \boldsymbol{i}\sin n\theta$ corresponds to the matrix
\begin{equation}\boldsymbol{\mathcal{R}}_n=\begin{pmatrix}\cos n\theta & -\sin n\theta\\ \sin n\theta & \cos n\theta\end{pmatrix}\end{equation}
which gives us the matrix form of one-dimensional RoPE.
A Brief Introduction to Quaternions
As mentioned above, the quaternion is a generalization of the complex number, and in fact it is also the "ancestor" of the matrix — historically, quaternions came before general matrix operations, and quaternions inspired much of matrix arithmetic. Some years ago I wrote posts such as The Quaternion, Deeply Intertwined with Vectors and The Geometry of Numbers and the Numbers of Geometry: A Brief Exploration of Hypercomplex Numbers introducing quaternions, which interested readers are welcome to check out.
If a complex number is a two-dimensional vector, then a quaternion is a four-dimensional vector, written as $a+b\boldsymbol{i}+c\boldsymbol{j}+d\boldsymbol{k}$, where $\boldsymbol{i}^2=\boldsymbol{j}^2=\boldsymbol{k}^2=-1$, but these are all mutually distinct, and the multiplication rules among the units are:
$$\begin{array}{c|cccc} \times & 1 & \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ \hline 1 & 1 & \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ \boldsymbol{i} & \boldsymbol{i} & -1 & \boldsymbol{k} & -\boldsymbol{j} \\ \boldsymbol{j} & \boldsymbol{j} & -\boldsymbol{k} & -1 & \boldsymbol{i} \\ \boldsymbol{k} & \boldsymbol{k} & \boldsymbol{j} & -\boldsymbol{i} & -1 \\ \end{array}$$
At the time, the biggest shock this brought to people was non-commutativity, e.g., $\boldsymbol{i}\boldsymbol{j}=-\boldsymbol{j}\boldsymbol{i}\neq \boldsymbol{j}\boldsymbol{i}$. But apart from that, quaternion arithmetic is in fact highly similar to complex-number arithmetic.
For instance, analogous to the complex Euler formula, we have
\begin{equation}e^{a+b\boldsymbol{i}+c\boldsymbol{j}+d\boldsymbol{k}} = e^a\left(\cos r + \frac{b\boldsymbol{i}+c\boldsymbol{j}+d\boldsymbol{k}}{r}\sin r\right)\label{eq:euler}\end{equation}
where $r = \Vert b\boldsymbol{i}+c\boldsymbol{j}+d\boldsymbol{k}\Vert = \sqrt{b^2+c^2+d^2}$. There's also an analogous matrix mapping:
\begin{equation} a+b\boldsymbol{i}+c\boldsymbol{j}+d\boldsymbol{k} \quad \leftrightarrow \quad \begin{pmatrix} a & -b & -c & -d \\ b & a & -d & c \\ c & d & a & -b \\ d & -c & b & a \end{pmatrix}\label{eq:mapping}\end{equation}
Violating Relativity
As for the origins behind these formulas, that's a long story I won't get into here — interested readers are welcome to look it up themselves. With the Euler formula and the exponential mapping in hand, some readers may have already jumped ahead: one-dimensional RoPE is essentially the matrix mapping of $e^{n\boldsymbol{i}\theta}$, so wouldn't mapping $e^{x\boldsymbol{i}\theta + y\boldsymbol{j}\theta}$ to matrix form give us two-dimensional RoPE?
That's exactly what I thought at first. Unfortunately, it's wrong. Where does it go wrong? In one-dimensional RoPE, we used the complex-number representation of the inner product:
\begin{equation}\langle\boldsymbol{q},\boldsymbol{k}\rangle=\text{Re}[\boldsymbol{q}\boldsymbol{k}^*]\end{equation}
This identity also holds for quaternions, so it can be carried over directly. Next we used the complex exponential:
\begin{equation}\langle\boldsymbol{q}e^{m\boldsymbol{i}\theta},\boldsymbol{k}e^{n\boldsymbol{i}\theta}\rangle=\text{Re}\left[\left(\boldsymbol{q}e^{m\boldsymbol{i}\theta}\right)\left(\boldsymbol{k}e^{n\boldsymbol{i}\theta}\right)^*\right]=\text{Re}\left[\boldsymbol{q}e^{m\boldsymbol{i}\theta}e^{-n\boldsymbol{i}\theta}\boldsymbol{k}^*\right]=\text{Re}\left[\boldsymbol{q}e^{(m-n)\boldsymbol{i}\theta}\boldsymbol{k}^*\right]\end{equation}
The first two equalities can also be carried over to quaternions directly. The crux of the matter is that the third equality does not hold in general for quaternions! In general, for two quaternions $\boldsymbol{p},\boldsymbol{q}$, the identity $e^{\boldsymbol{p}+\boldsymbol{q}}=e^{\boldsymbol{p}}e^{\boldsymbol{q}}$ does not hold. More broadly, for two objects whose multiplication is not commutative, in general $e^{\boldsymbol{p}+\boldsymbol{q}}\neq e^{\boldsymbol{p}}e^{\boldsymbol{q}}$.
So, by the end of the derivation, since exponentiated products cannot be converted into sums, the relativity property can no longer be guaranteed. So this quaternion-based route died right there...
Matrix Exponentials
The matrix representation of quaternions shows that quaternions in fact represent a specific family of $4\times 4$ matrices. If the quaternion route doesn't work, perhaps general matrix analysis will. And indeed it does — in this section we'll derive a result using the matrix exponential.
The Matrix Exponential
The matrix exponential here is not the elementwise exponential activation function used in neural networks, but rather the operation defined via a power series:
\begin{equation}\exp \boldsymbol{B} = \sum_{k=0}^{\infty}\frac{\boldsymbol{B}^k}{k!}\end{equation}
where $\boldsymbol{B}^k$ denotes the matrix product of $k$ copies of $\boldsymbol{B}$ multiplied together. I previously wrote a post on the matrix exponential, Appreciating the Identity det(exp(A)) = exp(Tr(A)), which readers are also welcome to check out.
The matrix exponential is an extremely important matrix operation; it lets us directly write down the solution of a constant-coefficient system of differential equations $\frac{d}{dt}\boldsymbol{x}_t=\boldsymbol{A}\boldsymbol{x}_t$:
\begin{equation}\boldsymbol{x}_t = \big(\exp t\boldsymbol{A}\big)\boldsymbol{x}_0\end{equation}
Though that's not really relevant to the topic of this post. For deriving RoPE, the property of the matrix exponential we mainly rely on is:
\begin{equation}\boldsymbol{A}\boldsymbol{B} = \boldsymbol{B}\boldsymbol{A} \quad\Rightarrow\quad \big(\exp \boldsymbol{A}\big)\big(\exp \boldsymbol{B}\big) = \exp \big(\boldsymbol{A} + \boldsymbol{B}\big)\label{eq:expm-ex}\end{equation}
That is, if $\boldsymbol{A},\boldsymbol{B}$ commute under multiplication, then the matrix exponential can convert products into sums just like ordinary scalar exponentials. Note, though, that this is a sufficient but not necessary condition.
As for how to actually compute a matrix exponential, that's beyond what we can cover here, but many software libraries already provide this operation — for instance, both the scipy and TensorFlow numerical libraries have an expm function, and for symbolic computation, Mathematica has a MatrixExp function.
The One-Dimensional General Solution
Why can RoPE be connected to the matrix exponential in the first place? Because one-dimensional RoPE has a fairly simple exponential expression:
\begin{equation}\boldsymbol{\mathcal{R}}_n=\begin{pmatrix}\cos n\theta & -\sin n\theta\\ \sin n\theta & \cos n\theta\end{pmatrix}=\exp\left\{n\theta\begin{pmatrix}0 & -1\\ 1 & 0\end{pmatrix}\right\}\label{eq:rope-exp}\end{equation}
This led me to consider matrices of the following form as candidate solutions for RoPE:
\begin{equation}\boldsymbol{\mathcal{R}}_n=\exp n\boldsymbol{B}\end{equation}
where $\boldsymbol{B}$ is a matrix independent of $n$. A necessary condition for RoPE is that it satisfy the "relativity" condition $\eqref{eq:re}$, so let's analyze
\begin{equation}\big(\exp m\boldsymbol{B}\big)^{\top}\big(\exp n\boldsymbol{B}\big) = \big(\exp m\boldsymbol{B}^{\top}\big)\big(\exp n\boldsymbol{B}\big)\end{equation}
Let's first assume that $\boldsymbol{B}^{\top},\boldsymbol{B}$ commute; then, by equation $\eqref{eq:expm-ex}$, we have
\begin{equation}\big(\exp m\boldsymbol{B}^{\top}\big)\big(\exp n\boldsymbol{B}\big) = \exp \big(m\boldsymbol{B}^{\top} + n\boldsymbol{B}\big)\end{equation}
For $m\boldsymbol{B}^{\top} + n\boldsymbol{B}=(n-m)\boldsymbol{B}$ to hold, it suffices that
\begin{equation}\boldsymbol{B}^{\top} = - \boldsymbol{B}\end{equation}
This is the constraint imposed by "relativity." We assumed just now that $\boldsymbol{B}^{\top},\boldsymbol{B}$ commute, and it's easy to check that any $\boldsymbol{B}^{\top},\boldsymbol{B}$ satisfying this equation does indeed commute, so the result is self-consistent.
This means that for any matrix $\boldsymbol{B}$ satisfying $\boldsymbol{B}^{\top} + \boldsymbol{B} = 0$, $\exp n\boldsymbol{B}$ is a solution of the equation $\eqref{eq:re}$, and moreover it can be shown that it must be an orthogonal matrix. In fact, based on $\exp n\boldsymbol{B}=\left(\exp \boldsymbol{B}\right)^n$, we can more directly state: for any orthogonal matrix $\boldsymbol{O}$, $\boldsymbol{\mathcal{R}}_n=\boldsymbol{O}^n$ is a solution of the equation $\eqref{eq:re}$.
For $2\times 2$ matrices, the general solution of $\boldsymbol{B}^{\top} + \boldsymbol{B} = 0$ is $\boldsymbol{B}=\begin{pmatrix}0 & -\theta\\ \theta & 0\end{pmatrix}$, which gives us the solution shown in equation $\eqref{eq:rope-exp}$.
The Two-Dimensional Constraint
Similarly, for two-dimensional RoPE, let's consider
\begin{equation}\boldsymbol{\mathcal{R}}_{x,y}=\exp \big(x\boldsymbol{B}_1 + y\boldsymbol{B}_2\big)\end{equation}
as a candidate solution. Repeating the derivation of the "relativity" condition above: first assume $x_1\boldsymbol{B}_1^{\top} + y_1\boldsymbol{B}_2^{\top}$ and $x_2\boldsymbol{B}_1 + y_2\boldsymbol{B}_2$ commute; then we obtain the following constraint conditions:
\begin{equation}\boldsymbol{B}_1^{\top} + \boldsymbol{B}_1 = 0,\quad \boldsymbol{B}_2^{\top} + \boldsymbol{B}_2 = 0\end{equation}
However, $x_1\boldsymbol{B}_1^{\top} + y_1\boldsymbol{B}_2^{\top}$ and $x_2\boldsymbol{B}_1 + y_2\boldsymbol{B}_2$ commuting means that $(\boldsymbol{B}_1,\boldsymbol{B}_1^{\top})$, $(\boldsymbol{B}_2,\boldsymbol{B}_2^{\top})$, $(\boldsymbol{B}_1,\boldsymbol{B}_2^{\top})$, and $(\boldsymbol{B}_2,\boldsymbol{B}_1^{\top})$ must all commute pairwise, but the two constraints above only guarantee that $(\boldsymbol{B}_1,\boldsymbol{B}_1^{\top})$ and $(\boldsymbol{B}_2,\boldsymbol{B}_2^{\top})$ commute, not the commutativity of the remaining pair. So we need to add this as an extra constraint, giving:
\begin{equation}\left\{\begin{aligned} &\boldsymbol{B}_1^{\top} + \boldsymbol{B}_1 = 0\\ &\boldsymbol{B}_2^{\top} + \boldsymbol{B}_2 = 0\\ &\boldsymbol{B}_1 \boldsymbol{B}_2^{\top} = \boldsymbol{B}_2^{\top} \boldsymbol{B}_1 \end{aligned}\right.\label{eq:2d-conds}\end{equation}
It's not hard to show that, given the first two conditions, this additional constraint is in fact equivalent to $\boldsymbol{B}_1 \boldsymbol{B}_2 = \boldsymbol{B}_2 \boldsymbol{B}_1$.
RoPE Emerges
Since $2\times 2$ matrices satisfying the first two conditions have only one independent parameter, they fail to satisfy "invertibility," so we need to consider at least $3\times 3$ matrices, which have 3 independent parameters:
\begin{equation}\begin{pmatrix}0 & -a & -b \\ a & 0 & -c \\ b & c & 0\end{pmatrix}\end{equation}
To guarantee invertibility, let's require $\boldsymbol{B}_1,\boldsymbol{B}_2$ to be "orthogonal" — for instance, let's set:
\begin{equation}\boldsymbol{B}_1=\begin{pmatrix}0 & -a & 0 \\ a & 0 & 0 \\ 0 & 0 & 0\end{pmatrix},\quad\boldsymbol{B}_2=\begin{pmatrix}0 & 0 & -b \\ 0 & 0 & -c \\ b & c & 0\end{pmatrix}\end{equation}
Without loss of generality we can also set $a=1$, and then solving condition $\eqref{eq:2d-conds}$ gives $b=0,c=0$, i.e., $\boldsymbol{B}_2$ can only be the all-zero solution, which doesn't meet our requirements. The Mathematica code for solving this is:
B[a_, b_, c_] = {{0, -a, -b}, {a, 0, -c}, {b, c, 0}};
B1 = B[1, 0, 0];
B2 = B[0, b, c];
Solve[{Dot[B1, B2] == Dot[B2, B1]}, {b, c}]
So we need to consider at least $4\times 4$ matrices, which have 6 independent parameters. Without loss of generality, consider the orthogonal decomposition:
\begin{equation}\boldsymbol{B}_1=\begin{pmatrix}0 & -a & -b & 0 \\ a & 0 & -c & 0 \\ b & c & 0 & 0 \\ 0 & 0 & 0 & 0\end{pmatrix},\quad\boldsymbol{B}_2=\begin{pmatrix}0 & 0 & 0 & -d \\ 0 & 0 & 0 & -e \\ 0 & 0 & 0 & -f \\ d & e & f & 0\end{pmatrix}\end{equation}
Solving gives
\begin{equation}d=cf,\quad e=-bf\end{equation}
Solving code:
B[a_, b_, c_, d_, e_,
f_] = {{0, -a, -b, -d}, {a, 0, -c, -e}, {b, c, 0, -f}, {d, e, f,
0}};
B1 = B[1, b, c, 0, 0, 0];
B2 = B[0, 0, 0, d, e, f];
Solve[{Dot[B1, B2] == Dot[B2, B1]}, {b, c, d, e, f}]
Notice that the result places no constraint on $f$, so for simplicity we can just set $f=1$ and let the remaining $b,c,d,e$ all be zero. In that case,
\begin{equation}\boldsymbol{\mathcal{R}}_{x,y}=\exp \,\begin{pmatrix}0 & -x & 0 & 0 \\ x & 0 & 0 & 0 \\ 0 & 0 & 0 & -y \\ 0 & 0 & y & 0\end{pmatrix}\end{equation}
We can add back the parameter $\theta$ and expand it out to arrive at:
\begin{equation}\boldsymbol{\mathcal{R}}_{x,y}=\exp \,\left\{\begin{pmatrix}0 & -x & 0 & 0 \\ x & 0 & 0 & 0 \\ 0 & 0 & 0 & -y \\ 0 & 0 & y & 0\end{pmatrix}\theta\right\}=\left( \begin{array}{cc:cc} \cos x\theta & -\sin x\theta & 0 & 0 \\ \sin x\theta & \cos x\theta & 0 & 0 \\ \hdashline 0 & 0 & \cos y\theta & -\sin y\theta \\ 0 & 0 & \sin y\theta & \cos y\theta \\ \end{array}\right)\end{equation}
A Further Note
That concludes the derivation of two-dimensional RoPE. Readers may naturally be wondering: how well does it work? Unfortunately, we don't yet have a complete set of experimental results — I hadn't worked on ViT-related tasks before, and this derivation of two-dimensional RoPE was only just finished, so progress has been slow. What I can say is that preliminary results suggest it's quite effective. Members of the EleutherAI team have also experimented with this scheme, and found it to perform better than existing alternative position encodings.
Since I've mentioned EleutherAI, let me say a bit more. EleutherAI is the team that made waves a while back with its stated goal of "reproducing GPT-3." After we proposed RoPE and RoFormer in Transformer Upgrade Path: 2. Rotary Position Embedding, Drawing on the Best of Many Approaches, we were fortunate enough to catch the attention of the EleutherAI team, who ran a lot of additional experiments confirming that RoPE outperforms many other position encodings (see their blog post Rotary Embeddings: A Relative Revolution). This encouraged us to write up the English paper RoFormer: Enhanced Transformer with Rotary Position Embedding and submit it to Arxiv. And the original question about two-dimensional RoPE actually came from the EleutherAI team in the first place.
Summary
This post presented our generalization of RoPE to two dimensions, taking "relativity" and "invertibility" as the starting points for determining the final form of two-dimensional RoPE. We tried two derivation routes — quaternions and matrix exponentials — and ultimately arrived at the final solution via the matrix exponential. Along the way, this derivation also helped deepen our own understanding of RoPE.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.