How to Scrape a Website: A Hands-On Tutorial for Scraping Baidu Baike

如何“扒”站？手把手教你爬百度百科～ 苏剑林 · 2017-05-17 · Machine Learning crawler corpus

I recently needed to scrape some children's story corpora for training word embeddings, so I found a few fairy-tale websites and scraped every story on them. Below I'll walk through this process using Python, and combine it with some of my earlier experience scraping Baidu Baike (Baidu's Wikipedia-like encyclopedia). This tutorial is suited to the following scenario: you need to crawl an entire specified website, and the target site has no anti-scraping measures. Under this premise, all that's really being tested is our traversal algorithm and coding skills.

Assumptions

Let's restate our assumptions clearly:

1. We need to traverse the entire website to scrape the information we want;
2. The website has no anti-scraping measures;
3. Every page on the site can eventually be reached by starting from the homepage and clicking through hyperlinks step by step.

What kind of websites satisfy this assumption? The answer is: quite a lot of them, including the story site we'll scrape below, as well as Baidu Baike, Hudong Baike, and so on.

Let's first look at how to scrape this story site:

http://wap.xigushi.com/

(For teaching purposes only, no malicious intent~)

Breadth-First Search

The breadth-first algorithm is actually quite simple: every time we crawl a page, we save all the internal hyperlinks found on that page, and then add any links that haven't already been queued into the queue.

That's it? That's it! It's that simple~ Note that a queue follows the "first in, first out" principle, so what I just described is indeed the breadth-first algorithm. Writing it in Python is also straightforward:

#! -*- coding:utf-8 -*-

import requests as rq
import re
import time
import codecs
from multiprocessing.dummy import Pool,Queue #dummy子库是多线程库
import HTMLParser
unescape = HTMLParser.HTMLParser().unescape #用来实现对HTML字符的转义

tasks = Queue() #链接队列
tasks_pass = set() #已队列过的链接
results = {} #结果变量
count = 0 #爬取页面总数

tasks.put('/index.html') #把主页加入到链接队列
tasks_pass.add('/index.html') #把主页加入到已队列链接

def main(tasks):
    global results,count,tasks_pass #多线程可以很轻松地共享变量
    while True:
        url = tasks.get() #取出一个链接
        url = 'http://wap.xigushi.com'+url
        web = rq.get(url).content.decode('gbk') #这里的编码要看实际情形而定
        urls = re.findall('href="(/.*?)"', web) #查找所有站内链接
        for u in urls:
            if u not in tasks_pass: #把还没有队列过的链接加入队列
                tasks.put(u)
                tasks_pass.add(u)
        text = re.findall('<article>([\s\S]*?)</article>', web)
        #爬取我们所需要的信息，需要正则表达式知识来根据网页源代码而写
        if text:
            text = ' '.join([re.sub(u'[ \n\r\t\u3000]+', ' ', re.sub(u'<.*?>|\xa0', ' ', unescape(t))).strip() for t in text]) #对爬取的结果做一些简单的处理
            results[url] = text #加入到results中，保存为字典的好处是可以直接以url为键，实现去重
        count += 1
        if count % 100 == 0:
            print u'%s done.'%count

pool = Pool(4, main, (tasks,)) #多线程爬取，4是线程数
total = 0
while True: #这部分代码的意思是如果20秒内没有动静，那就结束脚本
    time.sleep(20)
    if len(tasks_pass) > total:
        total = len(tasks_pass)
    else:
        break

pool.terminate()
with codecs.open('results.txt', 'w', encoding='utf-8') as f:
    f.write('\n'.join(results.values()))

In just a handful of lines, we've implemented a general-purpose, multithreaded, concurrent web-scraping framework. Much of this code follows fixed patterns and is highly reusable. This is the elegance of Python — as the saying goes, "life is short, I use Python."

Baidu Baike

The code above already gives us a general-purpose scraping framework. But suppose we want to scrape Baidu Baike or Hudong Baike — then we run into new problems. In our earlier code, all the data I/O was done entirely in memory, which is fine for a small site, but a site like an online encyclopedia with millions or even tens of millions of pages will overwhelm that approach. So we need to address two issues: 1) resumable crawling (being able to pick up where we left off); and 2) memory efficiency. As it turns out, both problems are solved by the same solution: a database. Earlier, we stored the queue in a Queue object and the results in a dictionary, both living in memory. If we move both of these into a database, both problems are naturally resolved.

As for databases, I personally like MongoDB. Setting aside its other merits, what strikes me most is how "Pythonic" it feels — when you use it together with pymongo, you barely feel like you're interacting with a database at all; it feels like you're just writing pure Python (by contrast, even when you use SQL through Python, you still basically can't avoid writing SQL statements). I won't go into how to install MongoDB here — let's assume it's already installed, along with pymongo. Here, then, is some reference code for scraping Baidu Baike:

#! -*- coding:utf-8 -*-

import requests as rq
import re
import time
import datetime
from multiprocessing.dummy import Pool
import pymongo #使用数据库负责存取
from urllib import unquote #用来对URL进行解码
from urlparse import urlparse, urlunparse #对长的URL进行拆分
import HTMLParser
unescape = HTMLParser.HTMLParser().unescape #用来实现对HTML字符的转移

pymongo.MongoClient().drop_database('baidubaike')
tasks = pymongo.MongoClient().baidubaike.tasks #将队列存于数据库中
items = pymongo.MongoClient().baidubaike.items #存放结果

tasks.create_index([('url', 'hashed')]) #建立索引，保证查询速度
items.create_index([('url', 'hashed')])

count = items.count() #已爬取页面总数
if tasks.count() == 0: #如果队列为空，就把该页面作为初始页面，这个页面要尽可能多超链接
    tasks.insert({'url':u'http://baike.baidu.com/item/科学'})

url_split_re = re.compile('&|\+')
def clean_url(url):
    url = urlparse(url)
    return url_split_re.split(urlunparse((url.scheme, url.netloc, url.path, '', '', '')))[0]

def main():
    global count
    while True:
        url = tasks.find_one_and_delete({})['url'] #取出一个url，并且在队列中删除掉
        sess = rq.get(url)
        web = sess.content.decode('utf-8', 'ignore')
        urls = re.findall(u'href="(/item/.*?)"', web) #查找所有站内链接
        for u in urls:
            try:
                u = unquote(str(u)).decode('utf-8')
            except:
                pass
            u = 'http://baike.baidu.com' + u
            u = clean_url(u)
            if not items.find_one({'url':u}): #把还没有队列过的链接加入队列
                tasks.update({'url':u}, {'$set':{'url':u}}, upsert=True)
 text = re.findall('<div class="content">([\s\S]*?)<div class="content">', web)
 #爬取我们所需要的信息，需要正则表达式知识来根据网页源代码而写

 if text:
 text = ' '.join([re.sub(u'[ \n\r\t\u3000]+', ' ', re.sub(u'<.*?>|\xa0', ' ', unescape(t))).strip() for t in text]) #对爬取的结果做一些简单的处理
 title = re.findall(u'<title>(.*?)_百度百科</title>', web)[0]
 items.update({'url':url}, {'$set':{'url':url, 'title':title, 'text':text}}, upsert=True)
            count += 1
            print u'%s, 爬取《%s》，URL: %s, 已经爬取%s'%(datetime.datetime.now(), title, url, count)

pool = Pool(4, main) #多线程爬取，4是线程数
time.sleep(60)
while tasks.count() > 0:
    time.sleep(60)

pool.terminate()

Sample output:

2017-05-17 20:20:18.428393, scraped "Biological Anthropology", URL: http://baike.baidu.com/item/体质人类学, total scraped so far: 167
2017-05-17 20:20:18.502221, scraped "Group Dynamics", URL: http://baike.baidu.com/item/群体动力学, total scraped so far: 168
2017-05-17 20:20:18.535227, scraped "Biological Taxonomy", URL: http://baike.baidu.com/item/生物分类学, total scraped so far: 169
2017-05-17 20:20:18.545897, scraped "Virology", URL: http://baike.baidu.com/item/病毒学, total scraped so far: 170
2017-05-17 20:20:18.898083, scraped "Chromatography (book title)", URL: http://baike.baidu.com/item/色谱法, total scraped so far: 171
2017-05-17 20:20:18.929467, scraped "Molecular Biology (natural science discipline)", URL: http://baike.baidu.com/item/分子生物, total scraped so far: 172
2017-05-17 20:20:18.974105, scraped "Geochemistry (discipline name)", URL: http://baike.baidu.com/item/地球化学, total scraped so far: 173
2017-05-17 20:20:18.979666, scraped "Nanotechnology (physics term)", URL: http://baike.baidu.com/item/纳米科技, total scraped so far: 174
2017-05-17 20:20:19.077445, scraped "Theoretical Chemistry", URL: http://baike.baidu.com/item/理论化学, total scraped so far: 175
2017-05-17 20:20:19.143304, scraped "Thermochemistry", URL: http://baike.baidu.com/item/热化学, total scraped so far: 176
2017-05-17 20:20:19.333775, scraped "Acoustics", URL: http://baike.baidu.com/item/声学, total scraped so far: 177
2017-05-17 20:20:19.349983, scraped "Mathematical Physics", URL: http://baike.baidu.com/item/数学物理, total scraped so far: 178
2017-05-17 20:20:19.662366, scraped "High Energy Physics", URL: http://baike.baidu.com/item/高能物理学, total scraped so far: 179
2017-05-17 20:20:19.797841, scraped "Physics (natural science discipline)", URL: http://baike.baidu.com/item/物理学, total scraped so far: 180
2017-05-17 20:20:19.809453, scraped "Condensed Matter Physics", URL: http://baike.baidu.com/item/凝聚态物理学, total scraped so far: 181
2017-05-17 20:20:19.898944, scraped "Atom (physics concept)", URL: http://baike.baidu.com/item/原子, total scraped so far: 182

This already implements the basic framework. Of course, when you use it yourself, you'll still need to adjust it to your needs — for example, strengthening the regular expressions to strip out useless boilerplate like "Favorite View my favorites 0 Helpful +1 Voted," or extracting structured information into separate storage, and so on (please don't be lazy and just use this as-is — the results will be full of noise~ there's no such thing as a free lunch). In short, feel free to build on this foundation however you like. After running the above code to completion, you can scrape roughly 2.8 million entries, which already covers most commonly used entries.

Reflections & Summary

Some readers might ask: doesn't Baidu Baike have links like http://baike.baidu.com/view/52650.htm? Couldn't you just iterate through the numbers to crawl the whole site?

If we were only concerned with Baidu Baike, this approach would indeed work. But it isn't general — Hudong Baike, for instance, has no such links. Since what we care about here is a general scraping approach, we stick with the breadth-first traversal strategy.

Once again, I want to stress: the websites mentioned and the code demonstrated in this article are for teaching purposes only, with no malicious intent whatsoever~