Enhancing Typecho's Search Functionality

增强typecho的搜索功能 苏剑林 · 2018-01-09 · Machine Learning website Python

科学空间 (kexue.fm) is built on the Typecho platform, and its sidebar comes with a built-in search box. However, Typecho's native search feature only does exact string matching, which means a lot of perfectly reasonable queries return nothing at all — for example, searching "2018 astronomical events" or "new word discovery algorithm" gives no results, simply because the exact strings don't appear anywhere in the articles.

This got me thinking about improving the search functionality, something a few readers had actually suggested before. I spent a couple of days looking into it. My original plan was to build a full-text search engine using the Python library Whoosh, but integrating it and maintaining it long-term felt like too much overhead, so I dropped that idea. Instead, I decided to just improve Typecho's own search mechanism, and with help from a (very capable) colleague at work, I got this improvement working.

Since this involves directly modifying Typecho's source files, any future Typecho upgrade could overwrite these changes, so I'm documenting it here for my own future reference.

Investigation

By searching on Github, I found that Typecho's search functionality is implemented in var/Widget/Archive.php, roughly around lines 1185–1192:more

        if (!$hasPushed) {
            $searchQuery = '%' . str_replace(' ', '%', $keywords) . '%';
            /**搜索无法进入隐私项保护归档 */

            $select->where('table.contents.password IS NULL')
            ->where('table.contents.title LIKE ? OR table.contents.text LIKE ?', $searchQuery, $searchQuery)
            ->where('table.contents.type = ?', 'post');
        }

As you can see, search results are returned by matching keywords in SQL, where % is the SQL wildcard character. This also reveals something interesting: if the query string itself contains spaces, those spaces get replaced with wildcards too, which makes the search a bit more flexible.

So a natural idea occurred to me: regardless of whether the query contains spaces, we could manually tokenize the query string ourselves and then join the resulting tokens with wildcards, achieving more flexible matching even without spaces in the original query. This was indeed the first approach I tried. However, the problem with this approach is: even after tokenization, results are only returned if all the tokens match — if even one token never appears anywhere in the blog, no results come back. To do better, we need each token to be treated as a candidate rather than a mandatory requirement.

Implementation

To achieve this, I wrote an HTTP interface in Python and deployed it on the server. This interface handles the tokenization and generates the SQL clause. Then I took the original code `$keywords = $this->request->filter('url', 'search')->keywords;替换为$keywords = $this->request->keywords;` and rewrote it as:

        if (!$hasPushed) {
            $url = 'http://127.0.0.1:7777/token?text=' . $keywords;
            $url = str_replace(' ', '%20', $url);
            $searchQuery = file_get_contents($url);

            /**当接口失效时使用简单全匹配 */
            if (!$searchQuery) {
                $searchQuery = 'SIGN(INSTR(table.contents.title, "' . $keywords . '"))';
                $searchQuery = $searchQuery . ' + SIGN(INSTR(table.contents.text, "' . $keywords . '"))';
            }

            /**搜索无法进入隐私项保护归档 */
            $select->where('table.contents.password IS NULL')
            ->where($searchQuery . ' > 0')
            ->where('table.contents.type = ?', 'post')
            ->order($searchQuery, Typecho_Db::SORT_DESC);
        }

where the interface http://127.0.0.1:7777/token?text= is a Python program:

#! -*- coding:utf-8 -*-

import bottle
import jieba
jieba.initialize()

def convert(s):
    ws = jieba.cut(s)
    search = []
    for i in ws:
        search.append('2*SIGN(INSTR(table.contents.title, "%s"))'%i)
        search.append('SIGN(INSTR(table.contents.text, "%s"))'%i)
    return '(%s)'%(' + '.join(search))

@bottle.route('/token', method='GET')
def token_home():
    text = bottle.request.GET.get('text')
    if not text:
        text = ''
    return convert(text)

if __name__ == '__main__':
    bottle.run(host='0.0.0.0', port=7777, server='gunicorn')

This interface returns the scoring portion of the SQL statement. The algorithm works as follows: first tokenize the query, then for each token, add 2 points if it appears in the article title, and 1 point if it appears in the article body, and sum everything up into a total score. The functions used, like SIGN and INSTR, can easily be looked up if you're not familiar with them. As a side note, I'd recommend the lightweight bottle library for writing HTTP interfaces in Python — it's extremely convenient.

One more thing needed modification: since our PHP-side changes use order($searchQuery, Typecho_Db::SORT_DESC); to sort results by score in descending order, this won't take effect automatically, because Typecho by default sorts everything by time in descending order. So we also need to modify lines 1396–1397 of the same file. The original code:

 $select->order('table.contents.created', Typecho_Db::SORT_DESC)
        ->page($this->_currentPage, $this->parameter->pageSize);

was changed to:

        if (strpos($select, 'INSTR') === false) {
            $select->page($this->_currentPage, $this->parameter->pageSize)
            ->order('table.contents.created', Typecho_Db::SORT_DESC);
        } else {
            $select->page($this->_currentPage, $this->parameter->pageSize);
        }

The basic idea is to check whether the current query is a search query: if it is, skip the time-based ordering; if not, sort by time as usual. We can't simply remove the time-ordering clause altogether, because that same line of code is also responsible for the homepage output, and the homepage listing must remain sorted by time.

Conclusion

Why use this hybrid Python-and-PHP approach instead of writing everything in pure PHP? Sure, a pure PHP solution would also be possible — there is in fact a PHP port of Jieba (结巴) for word segmentation. But the real issue is: I don't know PHP! And the PHP version of Jieba also requires extra configuration, which is a bit of a hassle. Using Python, on the other hand, is much simpler for me — if anything needs improving down the line, I just tweak the Python script.

Lastly, some might worry whether such a "brute-force" solution could run into performance issues. Indeed, if a blog had hundreds of thousands of articles, this approach would definitely run into serious efficiency problems. But for a blog with only a few hundred posts, that's simply not something worth worrying about.

Finally, I can search more freely now~ Feel free to share any further suggestions.