End-to-End Tencent CAPTCHA Recognition (46% accuracy)

端到端的腾讯验证码识别（46%正确率） 苏剑林 · 2016-12-14 · Machine Learning image deep learning OCR end-to-end

For the latest results, please see: http://kexue.fm/archives/4503/

A while back I was fortunate enough to receive from a fellow netizen a batch of labeled Tencent CAPTCHA samples (a sample of the CAPTCHA can be found at: http://captcha.qq.com/getimage), so I took some time to test out a CAPTCHA recognition model.

Tencent CAPTCHA

Samples

This batch of CAPTCHAs is fairly simple: 4 English letters, mixed upper- and lower-case, though case doesn't matter for the input. The characters are somewhat visually confusable, so a traditional segmentation-based approach would probably be quite difficult here. The end-to-end approach is: feed the CAPTCHA image directly into a few convolutional layers, then attach several classifiers (26-way each), and directly output the four letter labels. Honestly there isn't much to say about it — as long as you have samples, you can build it, and the framework is general enough to handle case-sensitive scenarios (52 classes) or mixed letter-and-digit scenarios (just add 10 more classes).more

That said, there's one aspect I found rather tricky: the labels are case-insensitive. In this batch of samples, all the labels are lowercase, but the letters appearing in the images can be either upper or lower case. This means that with a plain 26-way classification, we're forcing A and a into the same class — even though A and a can look quite different visually. Forcing this kind of grouping feels a bit like "asking too much of the model"... I suspect this is part of why the model's accuracy couldn't be pushed higher. But I don't have a great solution for it either.

Code

Without further ado, here's the code:

https://github.com/bojone/n2n-ocr-for-qqcaptcha

The model is very simple and quite conventional (it's just a single file — how complex could it be?).

It uses 4 convolutional layers to extract features from the image, and then connects this feature representation to 4 separate softmax heads, each a 26-way classifier. Note that you can't just conveniently use TimeDistributed here — TimeDistributed shares weights, but here we want to output different labels from the same feature using separate weights. If the weights were the same, wouldn't the outputs end up the same too? Also note that my Keras setup uses Theano as the backend, not TensorFlow, and the two handle image tensors differently, so readers using TensorFlow will need to adjust accordingly.

_____________________________________________________________
Layer (type) Output Shape Param # Connected to
=============================================================
input_15 (InputLayer) (None, 3, 129, 53) 0
_____________________________________________________________
convolution2d_40 (Convolution2D) (None, 32, 127, 51) 896 input_15[0][0]
_____________________________________________________________
maxpooling2d_48 (MaxPooling2D) (None, 32, 63, 25) 0 convolution2d_40[0][0]
_____________________________________________________________
convolution2d_41 (Convolution2D) (None, 32, 61, 23) 9248 maxpooling2d_48[0][0]
_____________________________________________________________
maxpooling2d_49 (MaxPooling2D) (None, 32, 30, 11) 0 convolution2d_41[0][0]
_____________________________________________________________
activation_37 (Activation) (None, 32, 30, 11) 0 maxpooling2d_49[0][0]
_____________________________________________________________
convolution2d_42 (Convolution2D) (None, 32, 28, 9) 9248 activation_37[0][0]
_____________________________________________________________
maxpooling2d_50 (MaxPooling2D) (None, 32, 14, 4) 0 convolution2d_42[0][0]
_____________________________________________________________
activation_38 (Activation) (None, 32, 14, 4) 0 maxpooling2d_50[0][0]
_____________________________________________________________
convolution2d_43 (Convolution2D) (None, 32, 12, 2) 9248 activation_38[0][0]
_____________________________________________________________
maxpooling2d_51 (MaxPooling2D) (None, 32, 6, 1) 0 convolution2d_43[0][0]
_____________________________________________________________
flatten_15 (Flatten) (None, 192) 0 maxpooling2d_51[0][0]
_____________________________________________________________
activation_39 (Activation) (None, 192) 0 flatten_15[0][0]
_____________________________________________________________
dense_63 (Dense) (None, 26) 5018 activation_39[0][0]
_____________________________________________________________
dense_64 (Dense) (None, 26) 5018 activation_39[0][0]
_____________________________________________________________
dense_65 (Dense) (None, 26) 5018 activation_39[0][0]
_____________________________________________________________
dense_66 (Dense) (None, 26) 5018 activation_39[0][0]
=============================================================
Total params: 48712
_____________________________________________________________

After several dozen rounds of training, the resulting model achieves accuracies of 0.89, 0.72, 0.73, and 0.87 for the 1st, 2nd, 3rd, and 4th characters respectively. Given this, the probability of getting all four characters correct should be

$$0.89\times0.72\times0.73\times0.87\approx 0.41$$

which works out to a 41% full-match rate. In actual testing, the result was even better, coming in at 46% — meaning roughly one out of every two images gets recognized correctly, which should already be quite practical in many scenarios. Of course, this accuracy is specific to this particular batch of samples; the real-world accuracy is likely somewhat lower, but I'd guess it's still above 10%? ^_^

I won't be releasing the training samples publicly, and I also can't readily share the model weights — if you need them, please contact me privately.

Afterword

According to the friend who gave me the samples, he's now hooked up to an API provided by someone else that achieves over 95% full-match accuracy. I was instantly filled with admiration, and would love to learn from whoever built that — but that program has already been commercialized, so I doubt I'll ever get a chance to study it. I imagine that person has been focused specifically on Tencent CAPTCHA recognition for a long time, unlike me, who spreads himself thin across many things without mastering any of them.

I'd welcome readers to share better modeling ideas — please feel free to offer your insights. The current model has fewer than 50,000 parameters, which might mean it's underfitting; I'll try to tune it more carefully when I get the chance.