The Transformer Positional Encodings That Have Kept Researchers Racking Their Brains
Unlike RNNs, CNNs, and other models, for Transformer models the addition of positional encoding is essential, because a pure attention module has no way of capturing the order of the input—that is, it cannot distinguish tokens at different positions. To address this, we broadly have two options: (1) find a way to fold positional information into the input, which is the general approach behind absolute positional encoding; (2) find a way to tweak the attention structure itself so that it becomes able to distinguish tokens at different positions, which is the general approach behind relative positional encoding.
Although this basically boils down to two major categories—absolute positional encoding and relative positional encoding—each category has in fact spawned all sorts of variants, and researchers have really racked their brains over this. There are also a number of positional encoding schemes that don't play by the rules at all. In this article, let's take a tour of the "eight immortals crossing the sea, each showing their own special skill" — the various encoding schemes researchers have devised to better express positional information.
Absolute Positional Encoding
Formally speaking, absolute positional encoding is the relatively simpler of the two approaches, but that hasn't stopped researchers from coming up with all kinds of clever ideas—there are quite a few variants here too. Generally speaking, absolute positional encoding is added directly to the input: at the $k$-th vector of the input, $\boldsymbol{x}_k$, a positional vector $\boldsymbol{p}_k$ is added to form $\boldsymbol{x}_k + \boldsymbol{p}_k$, where $\boldsymbol{p}_k$ depends only on the position index $k$. more
Trainable
Obviously, the most straightforward approach to absolute positional encoding is to not design anything special at all, and instead simply treat the positional encoding as trainable parameters. For example, with a maximum length of 512 and an encoding dimension of 768, we would initialize a $512\times 768$ matrix as the positional vectors and let it update as training proceeds. This is exactly the kind of positional encoding used by BERT, GPT, and other current models; in fact it can be traced back even further—for instance, Facebook's 2017 paper Convolutional Sequence to Sequence Learning already used it.
For this trainable form of absolute positional encoding, it's commonly held that its drawback is a lack of extrapolation ability: if the maximum pretraining length is 512, then the model can only handle sentences up to length 512, and anything longer simply can't be processed. Of course, one could also randomly initialize the positional vectors beyond 512 and then continue fine-tuning. But the author's recent research shows that, through a hierarchical decomposition approach, absolute positional encoding can actually be made to extrapolate to arbitrarily long ranges while retaining decent performance—see the earlier post Hierarchically Decomposed Positional Encoding Lets BERT Handle Ultra-Long Text for details. So in fact, lack of extrapolation isn't really an inherent drawback of absolute positional encoding either.
Trigonometric (Sinusoidal)
Trigonometric positional encoding, commonly known as Sinusoidal positional encoding, is an explicit solution proposed in Google's paper Attention is All You Need:
\begin{equation}\left\{\begin{aligned}&\boldsymbol{p}_{k,2i}=\sin\Big(k/10000^{2i/d}\Big)\\ &\boldsymbol{p}_{k, 2i+1}=\cos\Big(k/10000^{2i/d}\Big) \end{aligned}\right.\end{equation}
where $\boldsymbol{p}_{k,2i},\boldsymbol{p}_{k,2i+1}$ are respectively the $2i,2i+1$-th component of the encoding vector at position $k$, and $d$ is the dimensionality of the positional vector.
Clearly, the defining feature of trigonometric positional encoding is that it has an explicit generating rule, so one would expect it to have some degree of extrapolation ability. Another reason for using it is that, since $\sin(\alpha+\beta)=\sin\alpha\cos\beta+\cos\alpha\sin\beta$ and $\cos(\alpha+\beta)=\cos\alpha\cos\beta-\sin\alpha\sin\beta$, the vector at position $\alpha+\beta$ can be expressed as a combination of the vectors at positions $\alpha$ and $\beta$, which opens up the possibility of expressing relative positional information. But curiously, we now rarely see work that directly uses this exact form of absolute positional encoding—the reason is unclear.
Recursive
In principle, RNN models don't need positional encoding at all, since they inherently have the structural capacity to learn positional information (recursion effectively means the model can learn to "count"). So if we first pass the input through an RNN layer and then feed it into a Transformer, in theory no positional encoding would be needed. By the same logic, we could use an RNN model to learn a form of absolute positional encoding—for instance, starting from a vector $\boldsymbol{p}_0$ and obtaining the encoding vector at each position through a recursive scheme $\boldsymbol{p}_{k+1}=f(\boldsymbol{p}_k)$.
The ICML 2020 paper Learning to Encode Position for Transformer with Continuous Dynamical Model pushes this idea to its logical extreme: it proposes modeling positional encoding via a differential equation (ODE), $d\boldsymbol{p}_t/dt=\boldsymbol{h}(\boldsymbol{p}_t,t)$, in a scheme called FLOATER. FLOATER is clearly also a recursive model; the function $\boldsymbol{h}(\boldsymbol{p}_t,t)$ can be modeled with a neural network, so this kind of differential equation is also called a neural differential equation—work on this has gradually been picking up recently.
In theory, positional encodings based on recursive models should also have fairly good extrapolation ability, while also offering more flexibility than trigonometric positional encoding (for instance, it's easy to show that trigonometric positional encoding is just a particular special case of FLOATER). But clearly, recursive-form positional encoding sacrifices some parallelism, which may introduce a speed bottleneck.
Multiplicative
We mentioned earlier that the input $\boldsymbol{x}_k$ is generally combined with the absolute positional encoding $\boldsymbol{p}_k$ via $\boldsymbol{x}_k + \boldsymbol{p}_k$—but is there some "not-so-general" way of combining them? For instance, $\boldsymbol{x}_k \otimes \boldsymbol{p}_k$ (element-wise multiplication)? When we build models in practice, there are multiple ways to fuse two vectors—addition, multiplication, even concatenation are all viable options—so why does everyone default to only considering addition when it comes to absolute positional encoding?
Sadly, the author doesn't know the answer either. Perhaps people default to addition because vector addition carries a fairly clear geometric meaning—but for deep learning models, that geometric meaning doesn't really have much practical value. In a recent experiment the author came across, it seems that replacing "add" with "multiply"—i.e., $\boldsymbol{x}_k \otimes \boldsymbol{p}_k$—appears to give better results than $\boldsymbol{x}_k + \boldsymbol{p}_k$. The author hasn't done a thorough comparison, so this is just offered as a possibility. For the source of this experiment, see Research on Chinese Language Models: (1) Multiplicative Positional Encoding.
Relative Positional Encoding
Relative positional encoding doesn't fully model the position information of every input; instead, when computing attention, it takes into account the relative distance between the current position and the position being attended to. Since natural language generally relies more on relative position, relative positional encoding tends to perform quite well too. Relative positional encoding also offers much more flexibility, which really lets researchers' creativity run wild.
The Classic Form
Relative positional encoding originates from Google's paper Self-Attention with Relative Position Representations; Huawei's open-sourced NEZHA model also uses this kind of positional encoding, and basically all subsequent relative positional encoding variants are simple modifications following the same template.
It's generally believed that relative positional encoding was inspired by absolute positional encoding. Consider the general attention formulation with absolute positional encoding:
\begin{equation}\left\{\begin{aligned} \boldsymbol{q}_i =&\, (\boldsymbol{x}_i + \boldsymbol{p}_i)\boldsymbol{W}_Q \\ \boldsymbol{k}_j =&\, (\boldsymbol{x}_j + \boldsymbol{p}_j)\boldsymbol{W}_K \\ \boldsymbol{v}_j =&\, (\boldsymbol{x}_j + \boldsymbol{p}_j)\boldsymbol{W}_V \\ a_{i,j} =&\, softmax\left(\boldsymbol{q}_i \boldsymbol{k}_j^{\top}\right)\\ \boldsymbol{o}_i =&\, \sum_j a_{i,j}\boldsymbol{v}_j \end{aligned}\right.\end{equation}
where $softmax$ is normalized over the $j$ dimension, and all vectors here refer to row vectors. Let's first expand $\boldsymbol{q}_i \boldsymbol{k}_j^{\top}$:
\begin{equation} \boldsymbol{q}_i \boldsymbol{k}_j^{\top} = \left(\boldsymbol{x}_i + \boldsymbol{p}_i\right)\boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\left(\boldsymbol{x}_j + \boldsymbol{p}_j\right)^{\top} = \left(\boldsymbol{x}_i \boldsymbol{W}_Q + \boldsymbol{p}_i \boldsymbol{W}_Q\right)\left(\boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top}\right) \end{equation}
To introduce relative positional information, Google dropped the position term in the first item, and changed $\boldsymbol{p}_j \boldsymbol{W}_K$ in the second term into a binary positional vector $\boldsymbol{R}_{i,j}^{K}$, giving
\begin{equation} a_{i,j} = softmax\left(\boldsymbol{x}_i \boldsymbol{W}_Q\left(\boldsymbol{x}_j\boldsymbol{W}_K + \color{green}{\boldsymbol{R}_{i,j}^K}\right)^{\top}\right) \end{equation}
and also replaced $\boldsymbol{p}_j \boldsymbol{W}_V$ with $\boldsymbol{R}_{i,j}^{V}$ in $\boldsymbol{o}_i =\sum\limits_j a_{i,j}\boldsymbol{v}_j = \sum\limits_j a_{i,j}(\boldsymbol{x}_j\boldsymbol{W}_V + \boldsymbol{p}_j\boldsymbol{W}_V)$:
\begin{equation}\boldsymbol{o}_i = \sum_j a_{i,j}\left(\boldsymbol{x}_j\boldsymbol{W}_V + \color{green}{\boldsymbol{R}_{i,j}^{V}}\right) \end{equation}
The so-called "relative position" refers to changing the vector $\boldsymbol{R}_{i,j}^{K},\boldsymbol{R}_{i,j}^{V}$, which originally depends on the pair of coordinates $(i,j)$, so that it depends only on the relative distance $i-j$; and this is typically truncated so as to accommodate arbitrary distances:
\begin{equation}\begin{aligned} \boldsymbol{R}_{i,j}^{K} = \boldsymbol{p}_K\left[\text{clip}(i-j, p_{\min}, p_{\max})\right]\\ \boldsymbol{R}_{i,j}^{V} = \boldsymbol{p}_V\left[\text{clip}(i-j, p_{\min}, p_{\max})\right] \end{aligned}\label{eq:rp-clip}\end{equation}
This way, only a finite number of positional encodings are needed to express relative positions of any length (thanks to the truncation); regardless of whether $\boldsymbol{p}_K,\boldsymbol{p}_V$ is chosen to be trainable or trigonometric, it can meet the requirement of handling text of arbitrary length.
The XLNet Style
The XLNet-style positional encoding actually originates from the Transformer-XL paper Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context; it's just that because the XLNet model, which used the Transformer-XL architecture, outperformed BERT to some degree, Transformer-XL became widely known, and this kind of positional encoding usually ends up bearing XLNet's name instead.
XLNet-style positional encoding comes from fully expanding the above $\boldsymbol{q}_i \boldsymbol{k}_j^{\top}$:
\begin{equation} \boldsymbol{q}_i \boldsymbol{k}_j^{\top} = \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top} + \boldsymbol{p}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{p}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top}\label{eq:qk-exp} \end{equation}
Transformer-XL's approach is quite simple: directly replace $\boldsymbol{p}_j$ with the relative positional vector $\boldsymbol{R}_{i-j}$, and for the two instances of $\boldsymbol{p}_i$, just replace them with two trainable vectors $\boldsymbol{u},\boldsymbol{v}$:
\begin{equation}\boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\color{green}{\boldsymbol{R}_{i-j}^{\top}} + \color{red}{\boldsymbol{u}}\boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \color{red}{\boldsymbol{v}} \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\color{green}{\boldsymbol{R}_{i-j}^{\top}} \end{equation}
In this encoding scheme, $\boldsymbol{R}_{i-j}$ is not truncated as in equation $\eqref{eq:rp-clip}$; instead, it directly uses the Sinusoidal-style generation scheme. Since the encoding space of $\boldsymbol{R}_{i-j}$ isn't necessarily the same as that of $\boldsymbol{x}_j$, the $\boldsymbol{W}_K^{\top}$ in front of $\boldsymbol{R}_{i-j}$ is swapped for a separate matrix $\boldsymbol{W}_{K,R}^{\top}$; and $\color{red}{\boldsymbol{u}}\boldsymbol{W}_Q$ and $\color{red}{\boldsymbol{v}} \boldsymbol{W}_Q$ can be directly merged into single vectors $\color{red}{\boldsymbol{u}}$ and $\color{red}{\boldsymbol{v}}$, so the final formula used is:
\begin{equation}\boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_{K,R}^{\top}\color{green}{\boldsymbol{R}_{i-j}^{\top}} + \color{red}{\boldsymbol{u}}\boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \color{red}{\boldsymbol{v}} \boldsymbol{W}_{K,R}^{\top}\color{green}{\boldsymbol{R}_{i-j}^{\top}} \end{equation}
In addition, the positional bias on $\boldsymbol{v}_j$ is simply dropped, i.e., $\boldsymbol{o}_i = \sum\limits_j a_{i,j}\boldsymbol{x}_j\boldsymbol{W}_V$ is set directly. It seems that, starting from this work, subsequent relative positional encodings only get added to the attention matrix, and are no longer added to $\boldsymbol{v}_j$.
The T5 Style
The T5 model comes from the paper Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, which uses an even simpler relative positional encoding scheme. The idea still stems from the expansion $\eqref{eq:qk-exp}$: if we absolutely must interpret the meaning of each term, they can respectively be understood as "input–input," "input–position," "position–input," and "position–position" attention combined together. If we assume that content information and positional information should be independent (disentangled), then they shouldn't interact too much, so the "input–position" and "position–input" attention terms can be dropped; and $\boldsymbol{p}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top}$ is really just a scalar depending only on $(i,j)$, which we can directly train as a parameter, simplifying to:
\begin{equation}\boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \color{green}{\boldsymbol{\beta}_{i,j}}\end{equation}
In other words, this is nothing more than adding a trainable bias term on top of the attention matrix; and, just like in the XLNet style, the positional bias on $\boldsymbol{v}_j$ is simply dropped entirely. Microsoft's ICLR 2021 paper Rethinking Positional Encoding in Language Pre-training proposes TUPE positional encoding, which carries the same idea.
What's rather distinctive is that, unlike conventional positional encodings that treat $\boldsymbol{\beta}_{i,j}$ as a function of $i-j$ and truncate it, T5 applies a "bucketing" scheme to relative positions: a relative position of $i-j$ actually corresponds to position $f(i-j)$, following this mapping:
$$\begin{array}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c} \hline i - j & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15\\ \hline f(i-j) & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 8 & 8 & 8 & 9 & 9 & 9 & 9 \\ \hline i - j & 16 & 17 & 18 & 19 & 20 & 21 & 22 & 23 & 24 & 25 & 26 & 27 & 28 & 29 & 30 & \cdots\\ \hline f(i-j) & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 11 & 11 & 11 & 11 & 11 & 11 & 11 & 11 & \cdots \\ \hline\end{array}$$
For the exact mapping code, readers can just check the source. The idea behind this design is actually quite intuitive: for nearby positions (0–7), we need to distinguish them fairly finely, so each gets its own dedicated positional encoding; whereas for more distant positions (say 8–11), we don't need to distinguish them as precisely, so they can share a single positional encoding—and the farther away, the larger the range that can share one encoding, until we hit the specified cutoff and clip.
The DeBERTa Style
DeBERTa is also a Microsoft creation, released back in June of last year; the paper is DeBERTa: Decoding-enhanced BERT with Disentangled Attention. It's had a bit of a resurgence in popularity recently, partly because it was officially accepted at ICLR 2021, and partly because it topped the SuperGLUE leaderboard, edging out T5 by a small margin.
DeBERTa's main improvement is also in positional encoding, and it likewise starts from the expansion $\eqref{eq:qk-exp}$. Whereas T5 simply drops the 2nd and 3rd terms and keeps only the 4th term, replacing it with relative positional encoding, DeBERTa does the exact opposite—it drops the 4th term and keeps the 2nd and 3rd terms, replacing them with relative positional encoding (sure enough, research really is about enumerating all the permutations and combinations to see which one wins):
\begin{equation} \boldsymbol{q}_i \boldsymbol{k}_j^{\top} = \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\color{green}{\boldsymbol{R}_{i,j}^{\top}} + \color{green}{\boldsymbol{R}_{j,i}} \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} \end{equation}
As for the design of $\boldsymbol{R}_{i,j}$, it's truncated in the same way as in equation $\eqref{eq:rp-clip}$—nothing unusual there.
What's interesting about DeBERTa, though, is that it offers a fresh perspective on using relative versus absolute positional encoding. It points out that most NLP tasks probably only need relative positional information, but there genuinely are scenarios where absolute positional information helps more—so it splits the entire model into two parts to account for this. Taking the Base-size MLM pretraining model as an example: it has 13 layers in total, and the first 11 layers use only relative positional encoding—this part is called the Encoder—while the last 2 layers incorporate absolute positional information, a part it calls the Decoder, abbreviated as EMD (Enhanced Mask Decoder). For downstream task fine-tuning, the truncated model uses the 11-layer Encoder plus the 1-layer Decoder.
DeBERTa's strong SuperGLUE result validates its value, but the naming choices throughout the paper are honestly rather off-putting. For instance, its self-styled "Encoder" and "Decoder" terminology easily leads readers to mistakenly think this is a Seq2Seq model; and the abbreviation EMD happens to collide with Earth Mover's Distance. While name collisions are sometimes unavoidable, the names it collides with here are ones that everyone in the ML community is already quite familiar with, making the confusion all too easy—it's really hard to see what the authors were thinking...
Other Positional Encodings
Although absolute and relative positional encodings come in many flavors, they still fall within the classical range, and from the descriptions above we can clearly sense a strong pattern to them. Beyond these, there are also some approaches that don't follow the usual playbook at all, yet still manage to express positional information.
The CNN Style
Although the classic work applying CNNs to NLP, Convolutional Sequence to Sequence Learning, did add positional encoding into the mix, we know that typical CNN models—especially CNNs used for images—don't add any extra positional encoding at all. So how exactly does a CNN model capture positional information?
If asked, the author's guess would be that the anisotropy of convolutional kernels allows the model to distinguish relative positions in different directions. But the ICLR 2020 paper How Much Position Information Do Convolutional Neural Networks Encode? gives a possibly surprising answer: the positional information in CNN models leaks in through zero padding!
As we know, to keep the feature map size consistent during convolution, we typically pad the input with some zeros—and this paper shows that this operation gives the model the ability to recognize positional information. In other words, while the anisotropy of the convolution kernel does matter, the most fundamental factor is the presence of zero padding. One can imagine that what's actually being extracted is the relative distance between the current position and the padding boundary.
However, this capability relies on the locality of CNNs; a globally-connected, prior-free structure like attention doesn't lend itself to the same trick. For readers only interested in Transformer positional encoding schemes, consider this just a bit of extra background to broaden the view.
The Complex-Valued Style
Complex-valued positional encoding is perhaps the most unconventional positional encoding scheme of all; it comes from the ICLR 2020 paper Encoding word order in complex embeddings. The paper's core idea is to combine properties of complex numbers with some basic principles to derive its positional encoding form (Complex Order):
\begin{equation}\left[r_{j, 1} e^{\text{i}\left(\omega_{j, 1} k+\theta_{j, 1}\right)}, \ldots, r_{j, 2} e^{\text{i}\left(\omega_{j, 2} k+\theta_{j, 2}\right)}, \cdots, r_{j, d} e^{\text{i}\left(\omega_{j, d} k+\theta_{j, d}\right)}\right]\label{eq:complex}\end{equation}
Here $\text{i}$ is the imaginary unit, $j$ represents a given word, $k$ represents the position that word is at, and
\begin{equation}\begin{aligned} \boldsymbol{r}_j =&\, [r_{j, 1},r_{j, 2},\cdots,r_{j, d}]\\ \boldsymbol{\omega}_j =&\, [\omega_{j, 1},\omega_{j, 2},\cdots,\omega_{j, d}]\\ \boldsymbol{\theta}_j =&\, [\theta_{j, 1},\theta_{j, 2},\cdots,\theta_{j, d}]\\ \end{aligned}\end{equation}
represent three sets of word embeddings for word $j$. You read that right—it really does assume that each word has three separate sets of position-independent word embeddings (though of course some form of parameter sharing could reduce this to two sets or even one), and then the position-dependent embedding for position $k$ is computed via the formula above.
You think introducing multiple sets of word embeddings is its most unconventional feature? Not even close! Note that expression $\eqref{eq:complex}$ is still in complex form—guess what happens next? Convert it to real numbers? Nope—it's fed directly into a complex-valued model! In other words, this work goes all the way down the complex-valued model route: not only is the input embedding layer complex-valued, every single Transformer layer inside is complex-valued too, and the authors even implemented and compared complex-valued versions of FastText, LSTM, CNN, and other models! The first author of this paper is Benyou Wang, and if you search for his related work, it's essentially all centered around complex-valued models—truly a die-hard fan of the complex-valued model approach.
A Fused Scheme
As it happens, using complex numbers, the author has also come up with a rather clever positional encoding scheme that fuses absolute and relative positional encoding into one. It's shared here in the hope that interested readers will want to discuss and explore it further.
For simplicity, let's assume that $\boldsymbol{q}_m,\boldsymbol{k}_n$ are 2D row vectors located at positions $m,n$ respectively. Since they're 2D, we can treat them as complex numbers for the purposes of computation. As we know, the key operation in attention is the vector inner product, which in complex form can be written as:
\begin{equation}\langle \boldsymbol{q}_m, \boldsymbol{k}_n\rangle = \text{Re}\left[\boldsymbol{q}_m \boldsymbol{k}_n^*\right]\end{equation}
where $^*$ is the complex conjugate, the multiplication on the right-hand side is ordinary complex multiplication, and $\text{Re}[]$ denotes taking the real part of the result. The above equation says:
The inner product of two 2D vectors equals the real part of the product of one complex number with the conjugate of the other, when the vectors are viewed as complex numbers.
If we multiply $\boldsymbol{q}_m,\boldsymbol{k}_n$ by $e^{\text{i}m\theta},e^{\text{i}n\theta}$ respectively to get $\boldsymbol{q}_m e^{\text{i}m\theta}, \boldsymbol{k}_n e^{\text{i}n\theta}$, this is equivalent to equipping them with absolute positional encoding (since they now explicitly depend on the absolute positions $m,n$). Then, plugging them into the inner product, we get:
\begin{equation}\langle \boldsymbol{q}_m e^{\text{i}m\theta}, \boldsymbol{k}_n e^{\text{i}n\theta}\rangle = \text{Re}\left[\left(\boldsymbol{q}_m e^{\text{i}m\theta}\right) \left(\boldsymbol{k}_n e^{\text{i}n\theta}\right)^*\right] = \text{Re}\left[\boldsymbol{q}_m \boldsymbol{k}_n^* e^{\text{i}(m-n)\theta}\right]\end{equation}
What's rather delightful is that the inner product only depends on the relative position $m-n$! This elegantly fuses absolute and relative position together.
Note that we haven't gone as "wild" as Complex Order here—the computation above is still, in essence, confined to the real numbers; we're simply using complex numbers as a tool to carry out certain derivations. From the result above, we know that for a 2D real vector $[x,y]$ at position $n$, if we treat it as a complex number and multiply by $e^{\text{i}n\theta}$, we obtain the identity:
\begin{equation}(x + y\text{i})e^{\text{i}n\theta} = (x \cos n\theta - y\sin n\theta) + \text{i} (x \sin n\theta + y\cos n\theta)\end{equation}
This means that by using
\begin{equation}\begin{pmatrix}x \\ y\end{pmatrix} \to \begin{pmatrix}x \cos n\theta - y\sin n\theta \\ x \sin n\theta + y\cos n\theta \end{pmatrix} = \begin{pmatrix}x \\ y \end{pmatrix}\cos n\theta + \begin{pmatrix}-y \\ x \end{pmatrix}\sin n\theta\end{equation}
to endow $[x, y]$ with absolute positional information, the effect during the attention computation turns out to be equivalent to relative positional encoding. For vectors with more than two dimensions, we can consider grouping every two dimensions together and applying the same operation to each pair, with $\theta$ allowed to differ from group to group.
In this way, we obtain a positional encoding scheme that fuses absolute and relative position into one. In form it resembles multiplicative absolute positional encoding: by applying this encoding to $\boldsymbol{q},\boldsymbol{k}$, the resulting effect is equivalent to relative positional encoding, and if explicit absolute positional information is also needed, the same encoding can additionally be applied to $\boldsymbol{v}$. In short, through an operation defined purely in terms of absolute position, we can achieve both the effect of absolute position and the effect of relative position. Preliminary experiments suggest that it does work, though it hasn't been thoroughly validated yet—readers are welcome to try it out and discuss.
Summary
This article has surveyed a range of work on positional encoding, broadly divided into absolute, relative, and unconventional categories, through which we've seen all sorts of ingenious tricks. Finally, the author has shared a scheme of his own devising that fuses absolute and relative positional encoding, for the benefit of interested readers.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.