A Lightweight Deep Learning Tokenizer: NNCWS v0.1
Alright, I admit I've indulged in a bit of clickbait myself... The tokenizer described in this post is actually a three-layer neural network model, so it's really "shallow learning" rather than "deep learning" — but "deep learning" just sounds more appealing to write. NNCWS stands for Neural Network based Chinese Segment System, a neural-network-based Chinese word segmentation system. It's written in Python and fully open source; readers are welcome to try it out.
A Few Words First
What makes this program special? Almost nothing! This post simply combines a neural network with character embeddings to implement an n-grams-style tokenizer (7-grams are used in the code). It doesn't use a fancy model like the one in Chinese Word Segmentation Series 4: Character Tagging via seq2seq with Bidirectional LSTM, nor can it be trained unsupervised like in Chinese Word Segmentation Series 5: Unsupervised Segmentation Based on Language Models. This is purely a simple supervised model, trained on the 2014 People's Daily annotated corpus. more
So what's the point of this program? Two words: lightweight! Current deep learning programs tend to be quite bulky, requiring all sorts of dependency libraries — libraries that aren't always easy to install on certain platforms, such as Windows. On top of that, the parameter counts can be enormous, to the point where speed itself becomes an issue. As a result, a lot of deep learning work ends up being confined to the lab, at best yielding a paper or two, far from real production use. This program has been trimmed down as much as possible: the character embedding dimension and model size have both been shrunk, and in the end the whole thing was reimplemented in NumPy. In other words, I trained the model with Keras, extracted the model parameters, and then used NumPy to call these parameters directly — so the final program only needs NumPy to run. Hence its main feature: lightweight!
Of course, this kind of neural-network-based tokenizer already has some rudimentary semantic understanding, mainly reflected in its reasonably good handling of ambiguous word combinations, as well as decent performance in recognizing entities like personal names and place names. So it's still worth using in general scenarios. Since it was trained on the People's Daily corpus, it performs better on news-domain text.
Download and Usage
GitHub repository: https://github.com/bojone/NNCWS
First install the NumPy dependency, then run
git clone https://github.com/bojone/NNCWS.git
cd NNCWS
python
and you can use it right away
from nncws import NNCWS
mycut = NNCWS()
s = u'作为一个小国的领袖，卡斯特罗必然无法跟毛泽东等而观之。但是，在卡斯特罗身上，毕竟折射出那个伟大时代的光辉。 今天，我们向卡斯特罗告别，要告别的 是那个伟大的时代，但是我们要留下的，是那个时代不朽的精神内核： ——对理想的坚定追求。 ——对国家民族独立的坚定信念。 ——不怕威胁的“硬骨头”精神！'
print ' '.join(mycut.cut_words(s))
s = u'2000年1月，李彦宏创建了百度。经过十多年的发展，百度已经发展成为全球第二大独立搜索引擎和最大的中文搜索引擎。百度 的成功，也使中国成为美国、俄罗斯和韩国之外，全球仅有的4个拥有搜索引擎核心技术的国家之一。2005年，百度在美国纳斯达克成功 上市，并成为首家进入纳斯达克成分股的中国公司。百度已经成为中国最具价值的品牌之一。'
print ' '.join(mycut.cut_words(s))
which gives
As the leader of a small nation, Castro could hardly be placed on the same footing as Mao Zedong. But Castro embodied, after all, the brilliance of that great era. Today, as we bid farewell to Castro, what we are really bidding farewell to is that great era — but what we must preserve is the immortal spirit at the core of that era: — a steadfast pursuit of ideals. — an unwavering belief in national independence. — the "unyielding backbone" spirit that fears no threat!
In January 2000, Robin Li founded Baidu. After more than a decade of development, Baidu has grown into the world's second-largest independent search engine and the largest Chinese-language search engine. Baidu's success has also made China one of only four countries in the world — alongside the United States, Russia, and South Korea — to possess core search engine technology. In 2005, Baidu successfully listed on NASDAQ in the United States, becoming the first Chinese company to be included in the NASDAQ Composite Index. Baidu has become one of China's most valuable brands.
Training Process
If you're interested in the model architecture and training process, take a look at the file nncws_train.py — it contains the processing of the 2014 People's Daily corpus, along with the model structure and training code. The file is short and clear, so I won't go into further detail here. Training also requires Keras to be installed.
If you don't care about reading the code, just place the script in the directory containing the People's Daily corpus and run python nncws_train.py directly. The 2014 People's Daily corpus can easily be found and downloaded online.
A Few Final Words
Honestly, this is just a toy — hence version 0.1 — and it's probably still far from being truly practical. But it represents a new attempt: lowering the barrier to applying deep learning and related techniques (a high barrier to training doesn't matter much; what the general public cares about is the barrier to application), aiming to leverage the effectiveness of new models while retaining convenience.
An obvious drawback of this kind of model is that if a user finds the model doesn't meet their needs, there's no way for them to adjust it themselves. By contrast, with traditional dictionary-lookup-based methods, users can simply add entries to the dictionary — a much more flexible approach. So the next goal is to combine supervised training, unsupervised new-word discovery, and a vocabulary list all together. I already have some initial ideas on this — stay tuned.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.