Does High Local Cosine Similarity Guarantee High Global Cosine Similarity?
When analyzing a model's parameters, sometimes we treat all of the model's parameters as a single overall vector, and sometimes we split the parameters apart and look at them separately. For instance, a 7B-parameter LLAMA model has seven billion parameters — sometimes we treat this as "a single 7-billion-dimensional vector," sometimes we treat it, following the model's implementation, as "several hundred vectors of various dimensions," and in the most extreme case, we might even treat it as "seven billion 1-dimensional vectors." Given these different ways of viewing things, when we want to compute certain statistical quantities, there are correspondingly different ways to compute them — local computation versus global computation. This raises the question of how the locally computed metric relates to the globally computed one.
In this post we're concerned with the cosine similarity of two vectors. If the dimensions of two large vectors are split into several groups, and the cosine similarity of the corresponding sub-vectors within each group is large, does that guarantee the cosine similarity of the two large vectors is also large? The answer is no. Interestingly, this turns out to be connected to the famous "Simpson's paradox."
Background
This question arose from my analysis of how a change in an optimizer's parameter increment affects the change in the loss function. Specifically, suppose the optimizer's update rule is:
\begin{equation}\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \boldsymbol{u}_t\end{equation}more
where $\boldsymbol{u}_t$ is a vector along some specified update direction (or its negative). A first-order Taylor expansion then gives
\begin{equation}\mathcal{L}(\boldsymbol{\theta}_{t+1}) = \mathcal{L}(\boldsymbol{\theta}_t - \eta_t \boldsymbol{u}_t)\approx \mathcal{L}(\boldsymbol{\theta}_t) - \eta_t \langle\boldsymbol{u}_t,\boldsymbol{g}_t\rangle\end{equation}
Here $\boldsymbol{g}_t$ is the gradient $\nabla_{\boldsymbol{\theta}_t}\mathcal{L}(\boldsymbol{\theta}_t)$, so the change in the loss function is approximately
\begin{equation}- \eta_t \langle\boldsymbol{u}_t,\boldsymbol{g}_t\rangle = - \eta_t \Vert\boldsymbol{u}_t\Vert \Vert\boldsymbol{g}_t\Vert \cos(\boldsymbol{u}_t,\boldsymbol{g}_t)\end{equation}
This led me to want to look at the cosine similarity between $\boldsymbol{u}_t$ and $\boldsymbol{g}_t$, i.e., how well-aligned the update direction is with the gradient.
But here's the catch, as mentioned at the start of the post: a model's parameters can be split up in different ways. Should we treat all the model's parameters as one big vector and compute the cosine similarity between the update vector and the gradient globally, or should we compute it separately for each layer, each parameter, locally? I tried both, and after truncating the local cosine similarities (ensuring the cosine similarity between the update vector and the gradient is greater than some positive threshold for every parameter), I found that the global cosine similarity could actually fall below that threshold. This was rather surprising at first glance, so I did a bit of analysis.
A Simple Analysis
The question can be abstracted as follows:
Suppose the local cosine similarities of two vectors are all no less than $\lambda > 0$. Does that guarantee the global cosine similarity of the two vectors is also no less than $\lambda$?
As we already know, the answer is no. To disprove it we just need a counterexample: let $\boldsymbol{x}=(1,1),\boldsymbol{y}=(1,2)$. Clearly $\boldsymbol{x}\neq\boldsymbol{y}$, so $\cos(\boldsymbol{x},\boldsymbol{y})\neq 1$, yet their sub-vectors — i.e., each individual component — are all positive numbers, and as 1-dimensional vectors, positive numbers always have a cosine similarity of 1. So we've obtained a counterexample where every local cosine similarity is 1, yet the global similarity is less than 1.
For a more general analysis, let $\boldsymbol{x}=[\boldsymbol{x}_1,\boldsymbol{x}_2],\boldsymbol{y}=[\boldsymbol{y}_1,\boldsymbol{y}_2]$. Then
\begin{equation}\begin{aligned} \cos(\boldsymbol{x},\boldsymbol{y}) =&\, \frac{\langle \boldsymbol{x}, \boldsymbol{y}\rangle}{\Vert\boldsymbol{x}\Vert \Vert\boldsymbol{y}\Vert} = \frac{\langle \boldsymbol{x}_1, \boldsymbol{y}_1\rangle + \langle \boldsymbol{x}_2, \boldsymbol{y}_2\rangle}{\sqrt{\Vert\boldsymbol{x}_1\Vert^2 + \Vert\boldsymbol{x}_2\Vert^2} \sqrt{\Vert\boldsymbol{y}_1\Vert^2 + \Vert\boldsymbol{y}_2\Vert^2}} \\[6pt] =&\,\frac{\cos(\boldsymbol{x}_1, \boldsymbol{y}_1) \Vert\boldsymbol{x}_1\Vert \Vert\boldsymbol{y}_1\Vert+ \cos(\boldsymbol{x}_2, \boldsymbol{y}_2)\Vert\boldsymbol{x}_2\Vert \Vert\boldsymbol{y}_2\Vert}{\sqrt{\Vert\boldsymbol{x}_1\Vert^2 + \Vert\boldsymbol{x}_2\Vert^2} \sqrt{\Vert\boldsymbol{y}_1\Vert^2 + \Vert\boldsymbol{y}_2\Vert^2}} \end{aligned}\label{eq:cos}\end{equation}
If we let $\Vert\boldsymbol{x}_1\Vert,\Vert\boldsymbol{y}_2\Vert\to 0$, $\Vert\boldsymbol{x}_2\Vert,\Vert\boldsymbol{y}_1\Vert$ stay fixed and positive (without loss of generality we can set $\Vert\boldsymbol{x}_2\Vert=\Vert\boldsymbol{y}_1\Vert=1$), then we can get $\cos(\boldsymbol{x},\boldsymbol{y})\to 0$. That is, no matter how large $\cos(\boldsymbol{x}_1,\boldsymbol{y}_1)$ and $\cos(\boldsymbol{x}_2,\boldsymbol{y}_2)$ are, there's always a way to make $\cos(\boldsymbol{x},\boldsymbol{y})$ arbitrarily close to 0 — meaning $\cos(\boldsymbol{x}_1,\boldsymbol{y}_1)$ and $\cos(\boldsymbol{x}_2,\boldsymbol{y}_2)$ alone cannot give $\cos(\boldsymbol{x},\boldsymbol{y})$ a positive lower bound.
As for an upper bound, we can prove:
\begin{equation}\cos(\boldsymbol{x},\boldsymbol{y})\leq \max\big\{\cos(\boldsymbol{x}_1,\boldsymbol{y}_1),\cos(\boldsymbol{x}_2,\boldsymbol{y}_2)\big\}\label{eq:cos-ul}\end{equation}
The proof is actually quite simple, since this bound is rather loose. Without loss of generality, assume $\cos(\boldsymbol{x}_1,\boldsymbol{y}_1)\leq\cos(\boldsymbol{x}_2,\boldsymbol{y}_2)$. Then from equation $\eqref{eq:cos}$ we have
\begin{equation} \cos(\boldsymbol{x},\boldsymbol{y}) \leq\left[\frac{\Vert\boldsymbol{x}_1\Vert \Vert\boldsymbol{y}_1\Vert+ \Vert\boldsymbol{x}_2\Vert \Vert\boldsymbol{y}_2\Vert}{\sqrt{\Vert\boldsymbol{x}_1\Vert^2 + \Vert\boldsymbol{x}_2\Vert^2} \sqrt{\Vert\boldsymbol{y}_1\Vert^2 + \Vert\boldsymbol{y}_2\Vert^2}}\right]\cos(\boldsymbol{x}_2, \boldsymbol{y}_2) \end{equation}
The part in square brackets is precisely the cosine similarity of the two-dimensional vectors $(\Vert\boldsymbol{x}_1\Vert,\Vert\boldsymbol{x}_2\Vert)$ and $(\Vert\boldsymbol{y}_1\Vert,\Vert\boldsymbol{y}_2\Vert)$, so it must be no greater than 1, giving us $\cos(\boldsymbol{x},\boldsymbol{y})\leq\cos(\boldsymbol{x}_2,\boldsymbol{y}_2)$. This proves the inequality $\eqref{eq:cos-ul}$.
(Again, to emphasize: all the proofs above are carried out under the assumption that $\cos(\boldsymbol{x}_1,\boldsymbol{y}_1)\geq 0,\cos(\boldsymbol{x}_2,\boldsymbol{y}_2) \geq 0$. If negative values are allowed, the conclusions may need slight modification.)
A Related Paradox
Does this result have a more familiar real-world counterpart? Yes — applying it to correlation analysis leads directly to the famous "Simpson's paradox."
We know there's a coefficient for measuring linear correlation called the "Pearson Coefficient," defined as
\begin{equation}r = \frac{\sum\limits_{i=1}^n (x_i-\bar{x})(y_i - \bar{y})}{\sqrt{\sum\limits_{i=1}^n (x_i-\bar{x})^2}\sqrt{\sum\limits_{i=1}^n(y_i - \bar{y})^2}}\end{equation}
Looking more closely, if we let $\boldsymbol{x} = (x_1,x_2,\cdots,x_n),\boldsymbol{y} = (y_1,y_2,\cdots,y_n)$, isn't the expression above just
\begin{equation}r = \cos(\boldsymbol{x}-\bar{x},\boldsymbol{y}-\bar{y})\end{equation}
So the Pearson correlation coefficient is really just the cosine similarity computed after subtracting off the means from the data points. And once we've cast it as a cosine similarity, we can bring in the result from the previous section. The direct conclusion is that even if two batches of data are each clearly linearly correlated ($\cos > 0$), combining them may still result in no linear correlation at all ($\cos \to 0$).
"Simpson's paradox" goes even further, stating that even if every batch of data is positively correlated, combining them may not only eliminate the correlation but can even turn it negative. This is because the correlation coefficient has an extra $\bar{x},\bar{y}$ parameter compared to plain cosine similarity, giving it more tunable degrees of freedom. The geometric picture is also very intuitive, as shown in the figure below:
Illustration of Simpson's paradox
In the figure above, the blue data points all lie exactly on one line with positive slope, so their correlation coefficient is 1; the same is true of the red data — within its own batch it is "perfectly positively linearly correlated." But once we combine the data and insist on fitting a single line, we're forced into the dashed line, which has a negative slope — i.e., the correlation flips to negative. This is a classic illustration of Simpson's paradox.
Summary
This post briefly discussed the relationship between local and global cosine similarity for high-dimensional vectors, and further discussed the related "Simpson's paradox."
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.