Fetching and Processing the Chinese Wikipedia Corpus

获取并处理中文维基百科语料 苏剑林 · 2017-01-06 · Machine Learning wiki language corpus

Among Chinese-language corpora, the one that is both high-quality and easy to obtain is undoubtedly the Chinese Wikipedia corpus, and Wikipedia is remarkably generous, packaging up all its articles every month (download link here: https://dumps.wikimedia.org/zhwiki/) for anyone in the world to use—this is truly "taken from the people, given back to the people." Unfortunately, due to the unreasonable blocking imposed by the Chinese government, the Chinese Wikipedia currently has only about 910,000 articles, while Baidu Baike and Hudong Baike (互动百科) both have tens of millions of entries (and the English Wikipedia has well over ten million too). Even so, this hasn't stopped Chinese Wikipedia from being arguably the highest-quality Chinese-language corpus around. (Baidu Baike and Hudong Baike can only be obtained by crawling, and a lot of their entries are of pretty poor quality, largely just copied—or outright plagiarized—from one another.)

The Barrier to Entry

Downloading the dump is easy enough, but actually using the Wikipedia corpus does come with a certain barrier to entry. The raw downloaded corpus is a compressed text archive full of HTML and markdown markup, and basically can't be used as-is. Fortunately, some enterprising folks have already written processing tools for us. There are mainly two: 1. Wikipedia Extractor; 2. gensim's wikicorpus library. Both are Python-based.

However, neither of these two mainstream processing methods satisfies me. First, the output produced by Wikipedia Extractor strips out anything marked with {{}}, which leads to situations like the following:

The term "mathematics" in Western languages (; ) originates from ancient Greek (),

This happens because the words inside the parentheses were marked with {{}} and got wiped out entirely; and if you follow the online tutorials and process things directly with gensim.corpora.wikicorpus.WikiCorpus, the problem is even worse, because it strips out all punctuation as well. For someone as obsessive about getting a high-quality corpus as I am, neither of these is acceptable. So I wrote my own processing script, building on gensim.

Code

from gensim.corpora.wikicorpus import extract_pages,filter_wiki
import bz2file
import re
import opencc
from tqdm import tqdm
import codecs

wiki = extract_pages(bz2file.open('zhwiki-latest-pages-articles.xml.bz2'))

def wiki_replace(d):
    s = d[1]
    s = re.sub(':*{\|[\s\S]*?\|}', '', s)
    s = re.sub('<gallery>[\s\S]*?</gallery>', '', s)
    s = re.sub('(.){{([^{}\n]*?\|[^{}\n]*?)}}', '\\1[[\\2]]', s)
    s = filter_wiki(s)
    s = re.sub('\* *\n|\'{2,}', '', s)
    s = re.sub('\n+', '\n', s)
    s = re.sub('\n[:;]|\n +', '\n', s)
    s = re.sub('\n==', '\n\n==', s)
    s = u'【' + d[0] + u'】\n' + s
    return opencc.convert(s).strip()

i = 0
f = codecs.open('wiki.txt', 'w', encoding='utf-8')
w = tqdm(wiki, desc=u'已获取0篇文章')
for d in w:
    if not re.findall('^[a-zA-Z]+:', d[0]) and d[0] and not re.findall(u'^#', d[1]):
        s = wiki_replace(d)
        f.write(s+'\n\n\n')
        i += 1
        if i % 100 == 0:
            w.set_description(u'已获取%s篇文章'%i)

f.close()

Notes

As you can see, the main part of the code is regular expressions. First, bz2file is used to read the downloaded corpus without decompressing it, and then gensim's extract_pages is used to extract each page. After extraction, I first handle some of the page's special, non-text markup, then replace some of the useful {{}} markup with [[]], since [[]] markup doesn't get completely wiped out (readers will have to test the exact mechanics themselves), then clean things up directly with gensim's filter_wiki function, then handle line-break issues, and finally convert traditional characters to simplified characters via opencc.

In the loop that follows, the condition re.findall('^[a-zA-Z]+:', d[0]) removes help pages, and the condition re.findall(u'^#', d[1]) removes redirect pages, leaving roughly 919,000 pages in the end. tqdm is used to show progress, which is essential. The program took about 40 minutes to run on my machine, yielding roughly 1.5GB of plain-text corpus. The running time isn't really important, since preprocessing is a one-time cost.

One thing worth noting: opencc should not be installed via sudo apt-get install opencc, since the default version available that way is too old. You need to compile it from source, then run pip install opencc to install the Python interface. When you then try to call opencc from Python, you might get a "segmentation fault," in which case you need to run

cp /usr/lib/libopencc.so.1.0.0 /usr/lib/x86_64-linux-gnu/

A Side Product

Redirects, mentioned above, indicate that two terms share the same meaning. So I extracted all the redirects in Chinese Wikipedia and built a mapping table out of them. In other words, the two terms on each line of the resulting word list carry the same meaning. This turned out to be a nice side product.

A synonym table based on Chinese Wikipedia redirects: wiki_cn_mapping.7z