OCR: A Brief Exploration — 8. Overall Evaluation
Data Validation
Even though the model performs well in a test environment, practice remains the sole criterion of truth. In this section, we compare and validate our model against JD.com's test data.
Evaluating the quality of an OCR system involves two aspects: (1) whether it successfully draws boxes around the text; (2) for the text that has been boxed, whether it is correctly recognized. We use a scoring method to rate the recognition performance on each image. The scoring rule is as follows:
If a detected text region matches the box file in JD's provided detection samples, one point is added; if the text within is correctly recognized, another point is added. The final score for each image is the total score divided by the total number of text instances.
Under this rule, each image can score at most 2 points and at least 0. A score above 1 indicates fairly good recognition performance. After comparison against JD's test data, our model achieved an average score of about 0.84 — a passable but unremarkable result.more
Model Overview
In this article, our goal has been to build a complete OCR system, and through a series of efforts we have essentially accomplished this goal.
In designing the algorithm, we stayed closely aligned with our basic assumptions, starting from the idea of mimicking how the human eye recognizes text, hoping to achieve our goal with as few steps as possible. This philosophy is fully reflected in both the feature extraction and text localization components. Likewise, out of a preference for simplicity and for emulating human perception, we chose a convolutional neural network for optical character recognition, achieving fairly high accuracy; finally, by combining this with a language model, we used dynamic programming with a relatively simple approach to further improve performance.
Through testing, our system achieves good results in recognizing printed text, and can serve as an image-based text recognition tool for platforms such as e-commerce sites and WeChat. A notable feature is that our system can take an entire text image as input, and still achieve good results even at low resolution.
Reflections on the Results
A major shortcoming of the algorithms discussed in this article is that they rely on many "empirical parameters" — for example, the choice of the $h$ parameter during clustering, the density threshold used in defining low-density regions, the number of convolution kernels in the convolutional neural network, the number of hidden layer nodes, and so on. Since we lacked sufficiently large labeled datasets for systematic study, these parameters could only be estimated based on experience and a small number of samples. We hope that more labeled data will become available in the future to help determine the optimal values for these parameters.
Furthermore, there is much room for improvement in text region detection. Although we managed to filter out most non-text regions with just a few steps, these steps remain somewhat unintuitive and are in urgent need of simplification. We believe that a good model should be able to achieve strong results based on simple assumptions and procedures, so one direction worth pursuing is simplifying the assumptions and streamlining the pipeline.
In addition, when it comes to text segmentation, there is in fact no automatic segmentation algorithm capable of handling every situation, so this step still has considerable room for improvement. According to related literature, it is possible to directly recognize single lines of text using a CNN+LSTM model, but this requires a large number of training samples and high-performance training hardware — something that is likely feasible only for large enterprises.
Clearly, there is still much work that calls for deeper investigation.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.