Text Sentiment Classification (IV): A Better Loss Function

文本情感分类（四）：更好的损失函数 苏剑林 · 2017-03-30 · Machine Learning deep learning text mining loss function

Text sentiment classification is essentially a binary classification problem. In fact, all classification models share a common flaw: the optimization objective and the evaluation metric don't quite line up. Typically, for classification (including multi-class classification), we use cross-entropy as the loss function, which derives from maximum likelihood estimation (see Gradient Descent and the EM Algorithm: Two Sides of the Same Coin). But what we ultimately care about isn't how small the cross-entropy gets — it's the model's accuracy. Generally speaking, a small cross-entropy tends to come with high accuracy, but this relationship isn't strictly guaranteed.

Averages Don't Necessarily Need to Excel

A more down-to-earth analogy: a math teacher is working hard to raise the class average, but the metric used to evaluate the students at the end of term is the pass rate (a score of 60 or above counts as passing). If the average were 100 (meaning every student scored 100), then naturally the pass rate would be 100%, which is the ideal outcome. But reality isn't always that kind — as long as the average hasn't reached 100, a higher average doesn't necessarily mean a higher pass rate. For example, if two students score 40 and 90 respectively, the average is 65, but the pass rate is only 50%. If instead both students score 60, the average is only 60, but the pass rate is 100%. In other words, the average can serve as a training objective, but it isn't directly tied to the actual evaluation goal.
So, to actually improve the final evaluation metric, what should this teacher do? Clearly, the answer is to first check which students have already passed — leave them be — and focus remedial efforts on the students who haven't passed yet. In principle, this means many of the failing students can be brought up to 60, though some students who were previously passing might also slip below 60. But this process can be iterated, and eventually everyone ends up above 60. Of course, the final average might not be very high, but that's fine, since the evaluation metric is the pass rate, not the average.

A Better Update Scheme

For a binary classification model, we always want the model to output 1 for positive samples and 0 for negative samples. But due to limitations in the model's fitting capacity and so on, this generally can't be achieved perfectly. In practice, during prediction, we treat outputs above 0.5 as positive and below 0.5 as negative. This suggests we could update the model "selectively." For instance, set a threshold of 0.6: if the model's output for a positive sample already exceeds 0.6, we skip updating on that sample; if the model's output for a negative sample is already below 0.4, we likewise skip updating on that sample. Only samples with outputs between 0.4 and 0.6 trigger an update. This way, the model "focuses its energy" on the ambiguous samples, which should lead to better classification performance — an idea consistent with the classic SVM philosophy.

Not only that, but this approach should in theory also help prevent overfitting, since it stops the model from cherry-picking easy-to-fit samples to hammer away at (in order to drive the loss down further). It's analogous to a teacher who only cares about the top students, trying to push them from 80 to 90, while making no effort to help the weaker students improve — clearly not the mark of a good teacher.

A Corrected Cross-Entropy Loss

How do we achieve the goal described above? Simple — just adjust the loss function. The idea here borrows mainly from hinge loss and triplet loss. The commonly used cross-entropy loss function is:

$$L_{old} = -\sum_y y_{true} \log y_{pred}$$

Choose a threshold $m=0.6$, which in principle should be greater than 0.5. Introduce the unit step function $\theta(x)$:

$$\theta(x) = \left\{\begin{aligned}&1, x > 0\\ &\frac{1}{2}, x = 0\\ &0, x < 0\end{aligned}\right.$$

Then consider the new loss function:

$$L_{new} = -\sum_y \lambda(y_{true}, y_{pred}) y_{true}\log y_{pred}$$

where

$$\lambda(y_{true}, y_{pred}) = 1-\theta(y_{true}-m)\theta(y_{pred}-m)-\theta(1-m-y_{true})\theta(1-m-y_{pred})$$

What does $L_{new}$ mean, given that it's just the cross-entropy plus a correction term $\lambda(y_{true}, y_{pred})$? When a positive sample comes in, we have $y_{true}=1$, and clearly

$$\lambda(1, y_{pred})=1-\theta(y_{pred}-m)$$

In this case, if $y_{pred} > m$, then $\lambda(1, y_{pred})=0$, and the cross-entropy term automatically becomes 0 (its minimum value). Otherwise, if $y_{pred} < m$, then $\lambda(1, y_{pred})=1$, and the cross-entropy is left unchanged. In other words, if the output for a positive sample already exceeds $m$, the model no longer updates on it (since the minimum has been reached, and we can regard the gradient at the minimum as 0); only when the output is below $m$ does the update continue. Similarly, for negative samples: if the output is already below $1-m$, no update happens; only when it's above $1-m$ does the update continue.

In short, simply replacing the original cross-entropy loss with the corrected cross-entropy $L_{new}$ achieves exactly the goal we set out with.

Experimental Test on IMDB

The theory looks nice on paper — but does it hold up in practice? Let's find out.

To make the results more comparable, I chose a standard benchmark task in text sentiment classification: classifying IMDB movie reviews, using the latest version of Keras (2.0) as the toolkit. Most of the code can be found in Keras's examples, which include LSTM, CNN, and various other models.

First, the LSTM version:

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
from keras.datasets import imdb
from keras import backend as K

margin = 0.6
theta = lambda t: (K.sign(t)+1.)/2.

max_features = 20000
maxlen = 80
batch_size = 32

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)

x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)

model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

def loss(y_true, y_pred):
    return - (1 - theta(y_true - margin) * theta(y_pred - margin)
                - theta(1 - margin - y_true) * theta(1 - margin - y_pred)
             ) * (y_true * K.log(y_pred + 1e-8) + (1 - y_true) * K.log(1 - y_pred + 1e-8))

model.compile(loss=loss,
              optimizer='adam',
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=15,
          validation_data=(x_test, y_test))

The code is basically taken as-is from the official example, so no further explanation needed. After training, the model achieved a training accuracy of 99.01% and a test accuracy of 82.26%. If we switch the loss directly to binary_crossentropy (leaving everything else unchanged), we get 99.56% training accuracy and 81.02% test accuracy. This shows the new loss function does indeed help prevent overfitting and improve accuracy. Of course, this test may be subject to some random variance, but averaging over multiple runs still shows the new loss function typically brings a 0.5%–1% improvement in accuracy (though of course, merely tweaking the loss function slightly, you shouldn't expect any dramatic leap).

Now let's look at the CNN version:

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Embedding, Dense, Dropout, Activation
from keras.layers import Conv1D, GlobalMaxPooling1D
from keras.datasets import imdb
from keras import backend as K

margin = 0.6
theta = lambda t: (K.sign(t)+1.)/2.

max_features = 5000
maxlen = 400
batch_size = 32
embedding_dims = 50
filters = 250
kernel_size = 3
hidden_dims = 250
epochs = 10

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)

model = Sequential()
model.add(Embedding(max_features,
                    embedding_dims,
                    input_length=maxlen))
model.add(Dropout(0.2))
model.add(Conv1D(filters,
                 kernel_size,
                 padding='valid',
                 activation='relu',
                 strides=1))
model.add(GlobalMaxPooling1D())
model.add(Dense(hidden_dims))
model.add(Dropout(0.2))
model.add(Activation('relu'))
model.add(Dense(1))
model.add(Activation('sigmoid'))

def loss(y_true, y_pred):
    return - (1 - theta(y_true - margin) * theta(y_pred - margin)
                - theta(1 - margin - y_true) * theta(1 - margin - y_pred)
             ) * (y_true * K.log(y_pred + 1e-8) + (1 - y_true) * K.log(1 - y_pred + 1e-8))

model.compile(loss=loss,
              optimizer='adam',
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          validation_data=(x_test, y_test))

After training, the model achieved 98.66% training accuracy and 88.24% test accuracy. With plain binary_crossentropy, the results were 98.90% training accuracy and 88.14% test accuracy — essentially the same, within the range of normal fluctuation. However, during training, the test results using the new loss function stayed steady at around 88.2%, whereas with cross-entropy the numbers bounced around — jumping to 89%, then dropping to 87%, then back to 88%. In other words, although the final accuracies ended up similar, the cross-entropy training showed much more fluctuation. We have good reason to believe that the model trained with the new loss function generalizes better.

In Short

This post borrows ideas from hinge loss and triplet loss to adjust the cross-entropy loss used in binary classification, so that it focuses more effectively on fitting misclassified samples. The experiments show that, in some sense, this new loss function does bring a modest improvement.

This idea can, in fact, also be extended to multi-class classification and even regression problems, though I won't go into detailed examples here — I'll share more if and when I come across relevant cases worth discussing.