An NLP Library Based on the Minimum Entropy Principle: nlp zero

基于最小熵原理的NLP库：nlp zero 苏剑林 · 2018-05-31 · Machine Learning Python unsupervised minimum entropy

I've been writing a series of blog posts on the minimum entropy principle, aiming to lay some groundwork for unsupervised NLP. To make it easier for people to experiment, I've packaged the algorithms discussed in those posts into a library, for readers who want to try them out.

Since it's aimed at unsupervised NLP scenarios, and is basically foundational work for NLP tasks, I've named it nlp zero.

Links

GitHub: https://github.com/bojone/nlp-zero

PyPI: https://pypi.org/project/nlp-zero/

You can install it directly via

pip install nlp-zero==0.1.6

The whole library is implemented in pure Python with no third-party dependencies, and supports both Python 2.x and 3.x. more

Usage

Default tokenization

The library ships with a built-in dictionary, which can be used as a simple tokenization tool

from nlp_zero import *

t = Tokenizer()
t.tokenize(u'扫描二维码，关注公众号')

The bundled dictionary includes some new words mined through new-word discovery, and has been manually refined by me, so the quality is relatively high.

Building a vocabulary

Building a vocabulary from a large amount of raw corpus.

First we need to write an iterable container, so that we don't have to load the entire corpus into memory at once. The way you write the iterator is quite flexible — for example, if my data is stored in MongoDB, it would look like this:

import pymongo
db = pymongo.MongoClient().weixin.text_articles

class D:
    def __iter__(self):
        for i in db.find().limit(10000):
            yield i['text']

If the data is stored in a text file, it would roughly look like this:

class D:
    def __iter__(self):
        with open('text.txt') as f:
            for l in f:
                yield l.strip() # python2.x还需要转编码

Then we can run it:

from nlp_zero import *
import logging
logging.basicConfig(level = logging.INFO, format = '%(asctime)s - %(name)s - %(message)s')

f = Word_Finder(min_proba=1e-8)
f.train(D()) # 统计互信息
f.find(D()) # 构建词库

Inspect the results with Pandas:

import pandas as pd

words = pd.Series(f.words).sort_values(ascending=False)

Use the statistically derived vocabulary directly to build a tokenization tool:

t = f.export_tokenizer()

t.tokenize(u'今天天气不错')

Building sentence templates

As before, you need to write an iterator — I won't repeat that here.

Since sentence template construction is based on word-level statistics, we also need a tokenization function. You can use the built-in tokenizer, or an external one such as Jieba.

from nlp_zero import *
import logging
logging.basicConfig(level = logging.INFO, format = '%(asctime)s - %(name)s - %(message)s')

tokenize = Tokenizer().tokenize # 使用自带的分词工具
# 通过 tokenize = jieba.lcut 可以使用结巴分词

f = Template_Finder(tokenize, window=3)
f.train(D())
f.find(D())

Inspect the results with Pandas:

import pandas as pd

templates = pd.Series(f.templates).sort_values(ascending=False)
idx = [i for i in templates.index if not i.is_trivial()]
templates = templates[idx] # 筛选出非平凡的模版

Each template has already been wrapped as a class.

Hierarchical decomposition

Parsing sentence structure based on sentence templates.

from nlp_zero import *

# 建立一个前缀树，并加入模版
# 模版可以通过tuple来加入，
# 也可以直接通过“tire[模版类]=10”这样来加入
trie = XTrie()
trie[(None, u'呢')] = 10
trie[(None, u'可以', None, u'吗')] = 9
trie[(u'我', None)] = 8
trie[(None, u'的', None, u'是', None)] = 7
trie[(None, u'的', None, u'是', None, u'呢')] = 7
trie[(None, u'的', None)] = 12
trie[(None, u'和', None)] = 12

tokenize = Tokenizer().tokenize # 使用自带的分词工具
p = Parser(trie, tokenize) # 建立一个解析器

p.parse(u'鸡蛋可以吃吗') # 对句子进行解析
"""输出：
>>> p.parse(u'鸡蛋可以吃吗')
+---> (鸡蛋)可以(吃)吗
|     +---> 鸡蛋
|     |     +---> 鸡蛋
|     +---> 可以
|     +---> 吃
|     |     +---> 吃
|     +---> 吗
"""

To make it easier to work with the results and to visualize them, the output has been wrapped as a SentTree class. This class has three attributes: template (the current main template), content (the string covered by the current main template), and modules (a list of semantic chunks, each of which is itself described by a SentTree). In short, this design follows the assumptions about linguistic structure that I laid out in Minimum Entropy Principle (III): "Leapfrogging" — Sentence Templates and Linguistic Structure.

To be continued

If needed, please read the source code for answers~ Future updates will continue to be demonstrated here.