A Baseline for Fashion MNIST (MobileNet 95%)

fashion mnist的一个baseline (MobileNet 95%) 苏剑林 · 2017-08-27 · Machine Learning neural network deep learning

A First Taste

Yesterday I quickly tried a GAN model on Fashion MNIST and found that it actually works — though that experiment wasn't much of a technical feat, just tweaking the paths in an existing script and running it. Today I'm getting back to the main task of Fashion MNIST itself — 10-way classification. I tested a few models on it with Keras and eventually got an accuracy of around 94.5%, which reaches 95% with random-flip data augmentation on top.

At first I casually hand-crafted a few model combinations, but the accuracy turned out to be pretty mediocre across the board — it seems that for this dataset, designing a model from scratch is fairly hard. So I decided to use an off-the-shelf architecture instead. When it comes to ready-made CNN models, we usually think of VGG, ResNet, Inception, Xception, and the like, but these were designed for the 1000-class ImageNet problem, which seems overkill for this entry-level dataset — and also prone to overfitting. Then I suddenly remembered that Keras ships with a model called MobileNet. I checked its weight count and found it's small, but the capacity should still be decent, so I decided to run experiments with MobileNet.

Digging Deeper

I won't go into much detail about MobileNet here, since there are plenty of articles online explaining it. In short, it shares the same idea as Xception: it replaces most of the convolutions with depthwise convolutions. This depthwise convolution is a bit like the SVD decomposition of a matrix — it factorizes what would otherwise be a large convolution kernel matrix into two smaller matrices, ending up with fewer parameters while achieving even better results. A more recent piece of work along similar lines is ShuffleNet, but there's no Keras version yet, so I'll leave that aside for now.

The experiment is simple: load the MobileNet model, using the ImageNet pretrained weights by default (it's not obvious that ImageNet weights would help with this dataset, but they do indeed speed up convergence and improve accuracy — it seems many visual features are quite general-purpose), then attach a 10-way classifier head and train with all weights unfrozen. A couple of things to note:

1. MobileNet was originally designed for 224×224 inputs, while the Fashion MNIST images are only 28×28 — quite a big difference. Although feeding the images in directly wouldn't actually throw an error, I still upscaled the images by a factor of two, to 56×56, to avoid losing detail. You could of course scale them up even more, but there's no noticeable improvement in accuracy, just wasted compute.
2. MobileNet requires three-channel image input. To accommodate this, I simply duplicated the image across three channels.

The full code is as follows:

import numpy as np
import mnist_reader
from tqdm import tqdm
from scipy import misc
import tensorflow as tf

np.random.seed(2017)
tf.set_random_seed(2017)

X_train, y_train = mnist_reader.load_mnist('../data/fashion', kind='train')
X_test, y_test = mnist_reader.load_mnist('../data/fashion', kind='t10k')

height,width = 56,56

from keras.applications.mobilenet import MobileNet
from keras.layers import Input,Dense,Dropout,Lambda
from keras.models import Model
from keras import backend as K

input_image = Input(shape=(height,width))
input_image_ = Lambda(lambda x: K.repeat_elements(K.expand_dims(x,3),3,3))(input_image)
base_model = MobileNet(input_tensor=input_image_, include_top=False, pooling='avg')
output = Dropout(0.5)(base_model.output)
predict = Dense(10, activation='softmax')(output)

model = Model(inputs=input_image, outputs=predict)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()

X_train = X_train.reshape((-1,28,28))
X_train = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_train))])/255.

X_test = X_test.reshape((-1,28,28))
X_test = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_test))])/255.

model.fit(X_train, y_train, batch_size=64, epochs=50, validation_data=(X_test, y_test))

The code is simple and clear enough that I won't add comments.

After running the experiment multiple times, it basically reaches an accuracy above 94.5% within 20 epochs (although we set a random seed, due to cuDNN, the results still aren't guaranteed to be identical across runs). Beyond that, the later epochs become unstable, with some suspicion of overfitting.

A Closer Look

Without data augmentation, an accuracy above 94.5% already feels quite satisfying. I then tried adding data augmentation as well, but after some thought, there doesn't seem to be many augmentation techniques well suited to this dataset — the only one I could think of is random left-right flipping. Adding that in, here's the result:

import numpy as np
import mnist_reader
from tqdm import tqdm
from scipy import misc
import tensorflow as tf

np.random.seed(2017)
tf.set_random_seed(2017)

X_train, y_train = mnist_reader.load_mnist('../data/fashion', kind='train')
X_test, y_test = mnist_reader.load_mnist('../data/fashion', kind='t10k')

height,width = 56,56

from keras.applications.mobilenet import MobileNet
from keras.layers import Input,Dense,Dropout,Lambda
from keras.models import Model
from keras import backend as K

input_image = Input(shape=(height,width))
input_image_ = Lambda(lambda x: K.repeat_elements(K.expand_dims(x,3),3,3))(input_image)
base_model = MobileNet(input_tensor=input_image_, include_top=False, pooling='avg')
output = Dropout(0.5)(base_model.output)
predict = Dense(10, activation='softmax')(output)

model = Model(inputs=input_image, outputs=predict)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()

X_train = X_train.reshape((-1,28,28))
X_train = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_train))])/255.

X_test = X_test.reshape((-1,28,28))
X_test = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_test))])/255.

def random_reverse(x):
	if np.random.random() > 0.5:
		return x[:,::-1]
	else:
		return x

def data_generator(X,Y,batch_size=100):
	while True:
		idxs = np.random.permutation(len(X))
		X = X[idxs]
		Y = Y[idxs]
		p,q = [],[]
		for i in range(len(X)):
			p.append(random_reverse(X[i]))
			q.append(Y[i])
			if len(p) == batch_size:
				yield np.array(p),np.array(q)
				p,q = [],[]
		if p:
			yield np.array(p),np.array(q)
			p,q = [],[]		

model.fit_generator(data_generator(X_train,y_train), steps_per_epoch=600, epochs=50, validation_data=data_generator(X_test,y_test), validation_steps=100)

Sure enough, data augmentation does help somewhat. I ran it twice, getting 95.04% once and 94.91% the other time — meaning it can reach around 95% accuracy within 50 epochs. Note that not all augmentation techniques are helpful: I also tried adding random masking, and the accuracy actually dropped. So data augmentation needs to be tailored to the dataset — especially to the test set. Put bluntly, even though data augmentation is applied to the training set, its essence is really about injecting prior knowledge from the test set.

A Long Way to Go

It turns out Fashion MNIST really is quite challenging — unlike plain MNIST, where you can slap together a single Dense layer and casually get above 90% accuracy. This makes it a genuinely representative benchmark for CNN algorithms. Test accuracy on MNIST commonly exceeds 99%, whereas from what I've been able to find, the best results on Fashion MNIST are only around 96% (and even that without published source code). There's still a long way to go to reach 99% — it seems that even for a dataset like this, the road ahead is long indeed.

I wonder which model will be the first to reach 99% accuracy on it.