CoSENT (III): A Loss Function for Interactive Similarity Models
In CoSENT (I): A More Effective Sentence Embedding Scheme than Sentence-BERT], I proposed a supervised sentence embedding scheme called "CoSENT." Since it directly optimizes cosine similarity, it aligns more closely with the evaluation objective, and as a result it typically achieves better performance and faster convergence than Sentence-BERT. In CoSENT (II): How Big Is the Gap Between Feature-based Matching and Interactive Matching?], I also compared it against interactive similarity models, showing that on certain tasks it can even come close to matching interactive similarity models in performance.
Back then, however, my main goal was to find a replacement for Sentence-BERT that was closer to the evaluation objective, so all the results were framed around supervised sentence embeddings, i.e., feature-based similarity models. Recently it suddenly occurred to me that CoSENT could actually also serve as a loss function for interactive similarity models. So how does it compare with the standard choice of cross-entropy? This post supplements those experiments. more
Basic Recap
When CoSENT was first proposed, it was framed as a loss function for supervised sentence embeddings:
\begin{equation}\log \left(1 + \sum\limits_{\text{sim}(i,j) \gt \text{sim}(k,l)} e^{\lambda(\cos(u_k, u_l) - \cos(u_i, u_j))}\right)\end{equation}
Here $i,j,k,l$ are four training samples (say, four sentences), $u_i, u_j, u_k, u_l$ are the sentence vectors we want to learn (e.g., their [CLS] vectors after passing through BERT), $\cos(\cdot,\cdot)$ denotes the cosine similarity between two vectors, and $\text{sim}(\cdot,\cdot)$ denotes their similarity label. So the definition of this loss function is quite clear: whenever you believe the similarity of $(i,j)$ should be greater than the similarity of $\text{sim}(k,l)$, you add a term $e^{\lambda(\cos(u_k, u_l) - \cos(u_i, u_j))}$ into $\log$.
From this form, it's clear that CoSENT was originally designed for supervised training of feature-based models with cosine similarity—indeed, that's where the name "CoSENT" (Cosine Sentence) comes from. However, setting aside the cosine similarity aspect, CoSENT is fundamentally a loss function that depends only on the relative order of the labels; it has no inherent connection to cosine similarity. We can therefore generalize it as
\begin{equation}\log \left(1 + \sum\limits_{\text{sim}(i,j) \gt \text{sim}(k,l)} e^{\lambda(f(k,l) - f(i,j))}\right)\end{equation}
where $f(\cdot,\cdot)$ is any scalar-valued output function (generally no activation function is needed), representing the similarity model to be learned—including models that concatenate two inputs into a single text and feed it into BERT, i.e., an "interactive similarity" model!
Comparative Experiments
The conventional way to train an interactive similarity model is to construct a two-node output at the end, add a softmax, and use cross-entropy (abbreviated as CE in the table below) as the loss function. This is equivalent to adding a sigmoid activation on top of $f(\cdot,\cdot)$ and then using single-node binary cross-entropy. However, this approach is only suitable for binary-classification-style labels; for continuous scores (e.g., STS-B, which ranges from 1 to 5), it isn't a good fit, and one usually has to convert the problem into a regression task instead. CoSENT has no such restriction, since it only needs the ordinal information of the labels—a property that is consistent with the commonly used evaluation metric, the Spearman correlation coefficient.
For the reference code of the comparative experiment between the two, see:
https://github.com/bojone/CoSENT/blob/main/accuracy/interact_cosent.py]
The experimental results are as follows:
$$\begin{array}{c} \text{metric is spearman coefficient} \\ {\begin{array}{c|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{avg}\\ \hline \text{BERT + CE} & 48.01 & 71.96 & 78.53 & 68.59 & 66.77 \\ \text{BERT + CoSENT} & 48.09 & 72.25 & 78.70 & 69.34 & 67.10 \\ \hline \text{RoBERTa + CE} & 49.70 & 73.20 & 79.13 & 70.52 & 68.14 \\ \text{RoBERTa + CoSENT} & 49.82 & 73.09 & 78.78 & 70.54 & 68.06 \\ \hline \end{array}} \\ \\ \text{metric is accuracy} \\ {\begin{array}{c|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{avg}\\ \hline \text{BERT + CE} & 85.38 & 83.57 & 88.10 & 81.45 & 84.63 \\ \text{BERT + CoSENT} & 85.55 & 83.73 & 87.92 & 81.85 & 84.76 \\ \hline \text{RoBERTa + CE} & 85.97 & 84.67 & 88.14 & 82.85 & 85.41 \\ \text{RoBERTa + CoSENT} & 86.06 & 84.23 & 88.14 & 83.03 & 85.37 \\ \hline \end{array}} \end{array}$$
As we can see, there are no surprises here: CE and CoSENT perform essentially the same. If we try to dig out some finer-grained differences, we can observe that CoSENT performs slightly better with BERT, while with RoBERTa there's basically no difference; and on the PAWSX task, CoSENT's improvement is relatively more noticeable, while on the other tasks the two are essentially tied. Based on this, we can draw a "tentative" conclusion:
When the model is weaker (BERT being weaker than RoBERTa) or the task is harder (PAWSX being relatively harder than the other three tasks), CoSENT might achieve better results than CE.
Note the word "might"—I can't guarantee this either. To be honest, I don't think the two constitute any significant difference. That said, one could speculate that since the two loss functions differ noticeably in form, even if the final metrics end up similar, the models probably still differ internally to some extent—in which case, perhaps model ensembling could be worth considering?
Summary
This post mainly explored, both in thought and through experiments, the feasibility of using CoSENT in interactive similarity models. The conclusion is: "feasible, but with no real improvement in performance."
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.