General-Purpose Web Scraping (I): A Scraper for General Websites
This is a simplified version of the paper I submitted for this year's Teddy Cup Problem C competition. Although it only ended up receiving a consolation prize, I personally feel that some of the ideas in it are still worth sharing for anyone working on web scraping. So I'm putting it out here for reference.
Introduction
A web scraper can be broken down into two steps: 1. downloading the web page; 2. extracting the desired information from the page. Both steps have their own technical challenges. For the first step, the difficulty lies in dealing with the anti-scraping measures of various websites, such as IP bans or CAPTCHAs triggered by overly frequent access — this needs to be designed on a case-by-case basis for each website's particular anti-scraping measures, and in principle there is no truly general solution. For the second step, the traditional approach is to design corresponding regular expressions, but as website designs become increasingly diverse, writing such regular expressions becomes correspondingly harder.
Clearly, if we want a general-purpose scraping solution, relying on traditional regular expressions is quite difficult. But if we step outside the mindset constrained by regular expressions and instead look at websites from a global perspective, parsing them via the DOM tree, we can arrive at a fairly general solution. This post therefore focuses on the second step of scraping. The work here is divided into two parts: first, I propose an information extraction scheme applicable to general websites; then, I refine this scheme and apply it specifically to information extraction from forums. more
Workflow
This sounds fancy, but the underlying idea can actually be summed up in one sentence: "do subtraction" — page content minus page template equals the page's effective content. And where does the page template come from? It's the intersection of all pages on the same website!
The specific workflow is roughly as follows:
1. Download the pages of the target website and parse them into DOM trees;
2. Compare the DOM trees of several pages to obtain the site's base template;
3. Traverse the DOM tree in "depth-first" order, numbering each node and leaf of the tree;
4. Compare the DOM tree of each page against the standard template — the parts that differ are the effective text;
5. Apply some manual rules for further filtering;
6. Using the node and leaf numbers as keys, cluster and chunk the effective text.
Explanation
Let's elaborate a bit on the workflow above.
Ordinary web page source code is a mixture of HTML tags and effective text, and different websites use different templates, which is what makes general-purpose scraping difficult. However, within different pages of the same website, the pages share a certain uniformity. Put simply: on the same website, different pages tend to look pretty much alike, and the only real difference is the main content of the page.
Therefore, our design approach for a general-purpose scraper is: use different pages from the same website to construct a standard template for that website, then compare any page from that same website against the standard template — the parts that differ are the effective text we want to extract. In this process, we use the DOM tree to parse the syntax of the page source code.
HTML code
A reasonably well-formed HTML page's source code typically looks something like this:
<html>
<head>
<meta charset="UTF-8">
<title>这是网站的标题</title>
</head>
<body>
<div id="post_1">
<h2>这是文章的标题1</h2>
<a href="http://xxx.com"><img src="https://xxx.com/test.img"></a>
</div>
<div id="post_2">
<h2>这是文章的标题2</h2>
<a href="http://xxx.com"><img src="https://xxx.com/test.img"></a>
</div>
<div id="post_3">
<h2>这是文章的标题3</h2>
<a href="http://xxx.com"><img src="https://xxx.com/test.img"></a>
</div>
</body>
</html>
DOM tree
We can parse the source code of an HTML page into a tree structure called the DOM tree, short for Document Object Model. It is a document object model that uses an object-based representation to describe the structure of a document and its contents. The HTML page shown above, for instance, can be represented as the following DOM tree structure:
DOM tree
In other words, a web page is really just layers of nested HTML tags, where each tag is a node or leaf in the tree, and the page's text content sits inside these tags. One benefit of parsing into a DOM tree is that we can ignore interference from specific styling, for example:
<div id="content" class="content">
<h2 id="title_1" class="title_1">今天天气不错</h2>
<h2 id="title_2" class="title_2">通用爬虫方案</h2>
</div>
Here, div and h2 are both fixed tags, but in order to assign different styles to different headings, each <h2> tag has been given a different id and class. Since id and class can be set freely (or generated automatically according to some rule), this creates difficulty for general-purpose scraping. However, if we parse into a DOM tree, these specific markers are automatically discarded, leaving only the generic div and h2 tags.
Another benefit of parsing into a DOM tree is that it captures the hierarchical structure of the page, allowing us to traverse an HTML page in an orderly manner, and to partition the content we scrape according to the different levels of the page hierarchy.
Standard template
For a given website, we first download several pages one by one, then parse them into DOM trees. Next, we represent each node or leaf in the form "tag path + tag text." For the HTML page shown earlier, the site title can be represented as "html_head_title_This is the site title," and the article heading can be represented as "html_body_div_h2_This is article title 1." Note that not every tag has text content — for instance, an ![]() tag generally has no text — so here we only consider tags that carry text.
At this point, each web page is represented as a set of tags, and we can define the website's standard template as:
$$\begin{equation}S = \bigcap_{h\in H} h\end{equation}$$
where $H$ is the collection of all pages on the site. In other words, the standard template is the intersection of all pages. Extracting the effective text of a page is then simple:
$$\begin{equation}\text{effective} = h\backslash S\end{equation}$$
That is, the effective text of page $h$ is the set difference between set $h$ and set $S$.
With this approach, we can automatically construct a standard template for any given website, and thereby extract the effective text of its pages. This process rests on the assumption that "content shared between pages carries no information, while content that differs is worth scraping." Under this assumption, regardless of the site's structural hierarchy, and regardless of whether ads have been embedded, we can effectively filter out ineffective text. In this way, we obtain a general-purpose scraping scheme.
Pipelined generation
Although we define the standard template as the intersection of all pages, we don't actually need to scrape every single page before obtaining the standard template. At the start, we can use just two pages and take their intersection as the standard template, then update the standard template each time a new page is scraped, making the template progressively more accurate. This pipelined approach fits the requirements of a production environment.
Rule-based filtering
On some websites, different pages embed different pieces of code. As a result, scraping with the method above will also preserve these code snippets, reducing the precision of the effective text, so we need some filtering rules to remove them.
Engineering rule
From an engineering standpoint, one is usually willing to sacrifice some precision in exchange for greater speed. So the most obvious and easiest approach, from an engineering perspective, is to compute the ratio of Chinese to English character counts for each segment of effective text obtained above, and then, based on the assumption that "web page content is mostly Chinese," confidently discard the segments with a low proportion of Chinese characters.
Of course, this would also end up removing information like dates and usernames, so we first need to identify dates and usernames to avoid deleting them by mistake. Dates tend to follow relatively fixed patterns, such as "2017-04-15" or "April 15, 2017," and are often accompanied by words like "posted at" or "published on," making them relatively easy to identify. As for usernames, they are typically composed of Chinese characters, English letters, digits, and underscores, with no spaces allowed — and since code tends to contain a lot of spaces, this distinction lets us identify usernames in advance.
So, the overall rule is:
1. Text with a date-like format should be kept;
2. Text consisting purely of Chinese characters, English letters, digits, and underscores should be kept;
3. Among what remains, text whose Chinese-character proportion exceeds a certain threshold should be kept;
4. Everything else is filtered out.
Academic approach
The engineering rule is simple and efficient, but only applies to specific situations. For example, the rule above works well for scraping general Chinese-language websites, but wouldn't be suitable for English-language websites.
Broadly speaking, this is really a question of judging whether a piece of text is natural language, and we can use a language model for that. A language model models the probability of a sentence; given a sentence composed of $n$ words $w_1,w_2,\dots,w_n$, we consider the probability:
$$\begin{equation}P(w_1,w_2,\dots,w_n)\end{equation}$$
By the rules of probability theory, we obtain
$$\begin{equation}P(w_1,w_2,\dots,w_n)=P(w_1)P(w_2|w_1)P(w_3|w_1,w_2)\dots P(w_n|w_1,w_2,\dots,w_{n-1})\end{equation}$$
The right-hand side above is generally hard to estimate precisely, so we introduce a truncation: the probability of the next word depends only on the probability of the preceding word, which gives us
$$\begin{equation}\label{eq:score}P(w_1,w_2,\dots,w_n)=P(w_1)P(w_2|w_1)P(w_3|w_2)\dots P(w_n|w_{n-1})\end{equation}$$
This way, we only need to estimate the probability of each word $P(w_i)$ and the co-occurrence probability of adjacent words $P(w_{i-1}, w_{i})$, from which we can compute the transition probability
$$\begin{equation}P(w_i|w_{i-1}) = \frac{P(w_{i-1}, w_{i})}{P(w_{i-1})}\end{equation}$$
Once we have the transition probabilities, we can use equation $\eqref{eq:score}$ to score each piece of text, and then set a threshold: only text scoring above this threshold is judged to be natural language and thus kept. This approach works for all languages simultaneously (as long as a language model for the corresponding language is available), and is a theoretically more elegant model.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.