Tencent Captcha Recognition Based on Xception (with Samples and Code)

基于Xception的腾讯验证码识别（样本+代码） 苏剑林 · 2017-07-24 · Machine Learning image deep learning dataset

Last year, I was fortunate enough to get hold of a batch of Tencent captcha samples from a reader, and I did some research on it, which I documented in End-to-End Tencent Captcha Recognition (46% Accuracy).

That article later attracted quite a bit of interest from readers — some asking for the samples, some asking for the model, some wanting to discuss it further — which honestly caught me by surprise. To be fair, the original model was fairly rough, and its accuracy in particular was nothing to write home about, so it wasn't of much practical reference value. I've tinkered with it again over the past few days and put together a model with somewhat higher accuracy, and I'm also releasing the samples publicly this time.

The idea behind the model is the same as in End-to-End Tencent Captcha Recognition (46% Accuracy), except that the CNN part has been swapped out for the off-the-shelf Xception architecture. Of course, readers are welcome to try VGG, ResNet50, and so on instead — in fact, for captcha recognition, any of these models is up to the task. I chose Xception simply because it doesn't have too many layers and its weights are relatively small, which I happen to like.

Code

GitHub: https://github.com/bojone/n2n-ocr-for-qqcaptcha/

import glob
samples = glob.glob('sample/*.jpg')

import numpy as np
np.random.shuffle(samples) #打乱训练样本

nb_train = 90000 #共有10万样本，9万用于训练，1万用于测试
train_samples = samples[:nb_train]
test_samples = samples[nb_train:]

from keras.applications.xception import Xception,preprocess_input
from keras.layers import Input,Dense,Dropout
from keras.models import Model

img_size = (50, 120) #全体图片都resize成这个尺寸
input_image = Input(shape=(img_size[0],img_size[1],3))
base_model = Xception(input_tensor=input_image, weights='imagenet', include_top=False, pooling='avg')
predicts = [Dense(26, activation='softmax')(Dropout(0.5)(base_model.output)) for i in range(4)]

model = Model(inputs=input_image, outputs=predicts)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()

from scipy import misc
def data_generator(data, batch_size): #样本生成器，节省内存
    while True:
        batch = np.random.choice(data, batch_size)
        x,y = [],[]
        for img in batch:
            x.append(misc.imresize(misc.imread(img), img_size))
            y.append([ord(i)-ord('a') for i in img[-8:-4]])
        x = preprocess_input(np.array(x).astype(float))
        y = np.array(y)
        yield x,[y[:,i] for i in range(4)]

#训练过程终会显示逐标签的准确率
model.fit_generator(data_generator(train_samples, 100), steps_per_epoch=1000, epochs=10, validation_data=data_generator(test_samples, 100), validation_steps=100) 

#评价模型的全对率
from tqdm import tqdm
total = 0.
right = 0.
step = 0
for x,y in tqdm(data_generator(test_samples, 100)):
    _ = model.predict(x)
    _ = np.array([i.argmax(axis=1) for i in _]).T
    y = np.array(y).T
    total += len(x)
    right += ((_ == y).sum(axis=1) == 4).sum()
    if step < 100:
        step += 1
    else:
        break

print u'模型全对率：%s'%(right/total)

One thing worth noting: Xception's pretrained weights come from the ImageNet image classification task, which obviously isn't a good fit for captcha recognition. So here I've unfrozen all the layers for training, rather than freezing most of the weights as is typical in ordinary classification tasks.

Results

After training with the code above, the model achieves a recognition rate on the test set (counting a sample correct only if all four characters are correct) of over 85%. With finer-grained tuning — adjusting the learning rate, increasing or decreasing the number of iterations, tweaking the model architecture, and so on — it's possible to push this above 90%.

You might also want to check out the excellent work by Yang Peiwen, who used CTC for the final classification stage: Using Deep Learning to Crack Captcha.

Resources

100,000 captcha samples are available here:

Link: https://pan.baidu.com/s/1mhO1sG4 Password: j2rj