From Hard Truncation and Soft Weighting to Focal Loss

从loss的硬截断、软化到focal loss 苏剑林 · 2017-12-25 · Machine Learning model deep learning loss function

Preface

Today, in a discussion in a QQ group, I came across focal loss. After searching, I found it's a loss function proposed by Kaiming's team in their paper Focal Loss for Dense Object Detection, which they used to improve the performance of image object detection. However, I rarely work on image tasks, so I don't pay much attention to applications in that area. Fundamentally speaking, focal loss is a loss designed to address class imbalance and varying classification difficulty in classification problems, and in any case this work has received universal praise. You can also check out the discussion on Zhihu:

How would you evaluate Kaiming's Focal Loss for Dense Object Detection?

Upon seeing this loss, I initially found it quite fascinating and thought it could be very useful, because in NLP there are also plenty of tasks with severe class imbalance. The most classic example is sequence labeling tasks, where the classes are severely imbalanced — for instance, in named entity recognition, obviously in a given sentence there are far fewer entity tokens than non-entity tokens, which is a case of serious class imbalance. I tried applying it to my sequence-labeling-based question-answering model, and it did give a small improvement. Yes, this is indeed a good loss.

Then, after comparing it more carefully, I found that this loss shares a similar underlying idea with a loss I had come up with myself just last night! This is what prompted me to write this post. I'll approach the problem from my own line of thinking, analyze it, eventually arrive at focal loss, and also present the similar loss I derived last night. more

Hard Truncation

This entire post proceeds from the binary classification problem; the same ideas can be applied to multi-class classification. The standard loss for binary classification is cross-entropy:

$$L_{ce} = -y\log \hat{y} - (1-y)\log(1-\hat{y})=\left\{\begin{aligned}&-\log(\hat{y}),\,\text{if}y=1\\ &-\log(1-\hat{y}),\,\text{if}y=0\end{aligned}\right.$$

where $y\in\{0,1\}$ is the true label and $\hat{y}$ is the predicted value. Of course, for binary classification we almost always use the sigmoid function for activation $\hat{y}=\sigma(x)$, so this is equivalent to

$$L_{ce} = -y\log \sigma(x) - (1-y)\log\sigma(-x)=\left\{\begin{aligned}&-\log \sigma(x),\,\text{if}y=1\\ &-\log\sigma(-x),\,\text{if}y=0\end{aligned}\right.$$

(We have $1-\sigma(x)=\sigma(-x)$.)

In an earlier post from the first half of this year, Text Sentiment Classification (IV): A Better Loss Function, I once proposed a "hard truncation" loss based on the idea of "focusing on the hard-to-classify samples," of the form

$$L^\cdot = \lambda(y,\hat{y})\cdot L_{ce}$$

where

$$\lambda(y,\hat{y})=\left\{\begin{aligned}&0,\,(y=1\text{and}\hat{y} > 0.5)\text{or}(y=0\text{and}\hat{y} < 0.5)\\ &1,\,\text{other cases}\end{aligned}\right.$$

The idea here is: for positive samples whose predicted value is already greater than 0.5, or negative samples whose predicted value is already less than 0.5, we simply stop updating on them, and instead concentrate our attention on the samples that aren't predicted accurately yet — of course, this threshold could be adjusted. This approach can partially achieve the goal, but it requires far more iterations to converge.

The reason is this: take positive samples as an example — I'm only telling the model that once a positive sample's prediction exceeds 0.5, we stop updating on it, but I never tell it to "maintain" that state of being above 0.5. So in the next stage, its predicted value could easily drop back below 0.5 again. Of course, if that happens, the sample will get updated again in the next round, and this repeated iteration can theoretically still reach the goal — but the number of iterations required increases dramatically. So if we want to improve on this, the key point is: "it's not enough to just tell the model to stop updating once a positive sample's prediction exceeds 0.5 — we need to tell the model that once it exceeds 0.5, it just needs to maintain that." (It's like a teacher who stops paying attention to a student the moment they pass — that clearly doesn't work. If a student has already passed, we should find a way to help them maintain that level or even improve further, rather than ignoring them.)

Softening the Loss

The shortcoming of hard truncation lies precisely in the fact that the factor $\lambda(y,\hat{y})$ is non-differentiable — or rather, we treat its derivative as zero — so this term contributes nothing to the gradient, and consequently we can't get any meaningful feedback from it (in other words, the model has no way of knowing what "maintaining" means).

One way to solve this is to "soften" this loss. "Softening" means approximating some originally non-differentiable function with a differentiable one — mathematically this should really be called "smoothing." Once processed this way, what was originally non-differentiable becomes differentiable. A similar example can be found in the kmeans section of Gradient Descent and the EM Algorithm: Same Root, Same Lineage. Let's first rewrite $L^*$.

$$L^\cdot =\left\{\begin{aligned}&-\theta(0.5-\hat{y})\log(\hat{y}),\,\text{if}y=1\\ &-\theta(\hat{y}-0.5)\log(1-\hat{y}),\,\text{if}y=0\end{aligned}\right.$$

Here $\theta$ is just the unit step function

$$\theta(x) = \left\{\begin{aligned}&1, x > 0\\ &\frac{1}{2}, x = 0\\ &0, x < 0\end{aligned}\right.$$

This form of $L^*$ is completely equivalent to the original, and it's also equivalent to (since $\sigma(0)=0.5$)

$$L^\cdot =\left\{\begin{aligned}&-\theta(-x)\log \sigma(x),\,\text{if}y=1\\ &-\theta(x)\log\sigma(-x),\,\text{if}y=0\end{aligned}\right.$$

At this point the idea becomes clear: to "soften" this loss, we need to "soften" $\theta(x)$, and there's nothing easier to soften it with than the sigmoid function! We have

$$\theta(x) = \lim_{K\to +\infty} \sigma(Kx)$$

So evidently, all we need to do is replace $\theta(x)$ with $\sigma(Kx)$:

$$L^{\cdot \cdot }=\left\{\begin{aligned}&-\sigma(-Kx)\log \sigma(x),\,\text{if}y=1\\ &-\sigma(Kx)\log\sigma(-x),\,\text{if}y=0\end{aligned}\right.$$

This is exactly the loss I thought up last night, and it's clearly quite easy to implement.

Now let's compare it with focal loss.

Focal Loss

Kaiming's focal loss takes the form

$$L_{fl}=\left\{\begin{aligned}&-(1-\hat{y})^{\gamma}\log \hat{y},\,\text{if}y=1\\ &-\hat{y}^{\gamma}\log (1-\hat{y}),\,\text{if}y=0\end{aligned}\right.$$

If we plug in the prediction $\hat{y}=\sigma(x)$, then we have

$$L_{fl}=\left\{\begin{aligned}&-\sigma^{\gamma}(-x)\log \sigma(x),\,\text{if}y=1\\ &-\sigma^{\gamma}(x)\log\sigma(-x),\,\text{if}y=0\end{aligned}\right.$$

In particular, if $K$ and $\gamma$ are both set to 1, then $L^{**}=L_{fl}$!

In fact, $K$ and $\gamma$ play the same role — both adjust the steepness of the weighting curve, just through different mechanisms. Note that $L^{**}$ or $L_{fl}$ already inherently accounts for the problem of class imbalance — or, put another way, class imbalance is fundamentally just a manifestation of differing classification difficulty. For example, if there are far more negative samples than positive ones, the model will naturally tend to favor the majority negative class (imagine all samples being classified as negative). In that case, $\hat{y}^{\gamma}$ or $\sigma(Kx)$ for the negative class will be small, while $(1-\hat{y})^{\gamma}$ or $\sigma(-Kx)$ for the positive class will be large, causing the model to start focusing its attention on the positive samples.

Of course, Kaiming's team also found that applying an additional weight adjustment to $L_{fl}$ yields a small further improvement:

$$L_{fl}=\left\{\begin{aligned}&-\alpha(1-\hat{y})^{\gamma}\log \hat{y},\,\text{if}y=1\\ &-(1-\alpha)\hat{y}^{\gamma}\log (1-\hat{y}),\,\text{if}y=0\end{aligned}\right.$$

Through a series of hyperparameter tuning experiments, they found that $\alpha=0.25,\gamma=2$ (in their model) worked best. Note that in their task, positive samples are the minority class — in other words, positive samples originally struggle to "compete" with negative samples, but after being "manipulated" by $(1-\hat{y})^{\gamma}$ and $\hat{y}^{\gamma}$, the situation may actually flip, requiring the positive samples to be down-weighted again. However, I think this kind of adjustment is purely an empirical result — theoretically, it's hard to have a principled way of deciding the value of $\alpha$. If you don't have massive computational resources for hyperparameter tuning, you might as well just set $\alpha=0.5$ (equal weighting).

Multi-Class Case

The multi-class form of focal loss is easy to derive as well — it's simply

$$L_{fl}=-(1-\hat{y}_t)^{\gamma}\log \hat{y}_t$$

$\hat{y}_t$ is the predicted value for the target class, generally the result after applying softmax. So how do we generalize my own $L^{**}$ to the multi-class setting? That's also straightforward:

$$L^{\cdot \cdot }=-\text{softmax}(-Kx_t)\log \text{softmax}(x_t)$$

Here $x_t$ is also the predicted value for the target class, but it's the value before applying softmax.

Conclusion

**What? You came up with the same idea as Kaiming's team? No, no, no — this post is really just an introduction to Kaiming's focal loss, or more precisely, an introduction to some approaches for handling class imbalance and varying classification difficulty, together with my own thoughts on the matter as far as possible. Of course, writing it this way risks coming across as jumping on the bandwagon or crudely imitating something far superior — I ask the reader's indulgence.