Cool Papers Update: A Simple Site Search System
Since "A More Convenient Way to Open Cool Papers: A Chrome Redirect Extension", Cool Papershas undergone two fairly major changes. One was the introduction of a venue tab, which has gradually collected paper sets from various conferences over the years, such as ICLR and ICML — this part is dynamically and manually expanded, so readers with a favorite conference in mind are welcome to submit more requests. The other is the subject of this post: the site-wide search feature that was added the day before yesterday.
This post will give a brief introduction to the new feature, and offer a basic summary of the process of building the site search system.
Introduction
On the Cool Papers homepage, you'll see the search entry point:
Cool Papers（2024.05.07）more
The features of the search function are as follows:
1. It only searches the title and summary fields; specifying other fields is not yet supported.
2. You can specify searching either the arxiv tab or the venue tab, but mixed searches across both tabs are not supported.
3. Special characters in the search query (anything other than English letters and digits) will be stripped out.
4. Words in the search query are not automatically stemmed, meaning that searching for "images" will not match "image".
5. On the search results page, this can be used together with the original in-page search function.
Overall, this is currently just a very simple text search feature, meant to meet the basic needs of some users. For more complex needs, further updates will follow gradually. Features that might be gradually introduced in the future include specifying fields, searching Kimi FAQ content, sorting by stars, specifying dates/categories (for arxiv), specifying conferences (for venue), and even enabling addition/subtraction operations like ordinary search engines (e.g., excluding certain keywords). These will depend on user feedback going forward — there's no fixed schedule.
Summary
In fact, the need for a site search feature was raised by users back at the start of the year when Cool Papers was first opened to the public. The main reason it took so long to roll out was that Cool Papers collects papers day by day, and at the very beginning the number of papers was small, so site search didn't make much sense yet. After more than four months of accumulation, the number of Arxiv papers indexed by Cool Papers has exceeded 80,000, and together with the conference papers in the venue tab, which also number over 80,000, there are now nearly 170,000 papers in total — enough to be worth searching through.
Once it was decided that this was worth doing, the next question was how to do it. A retrieval system based on keyword search over document content is generally called "full-text search," and is typically built on inverted indexes and BM25 similarity — that is, the algorithmic side is well established. On the implementation side, Cool Papers' backend is built with BottlePy, so we needed to find a full-text search library available in Python that could be conveniently integrated into Cool Papers.
The options for "Python + Full-text Search" are not plentiful. The most classic one is a library called Whoosh, and functionally it does indeed meet Cool Papers' needs. But the problem with Whoosh is that it hasn't been updated since April 2016, which always leaves one a bit worried about hidden risks. Another option would be to switch directly to a database with built-in full-text search capabilities to store the data, such as MongoDB. If Cool Papers had used MongoDB from the start, that would undoubtedly have been the simplest solution, but Cool Papers instead chose Shelve, Python's built-in key-value database. Switching to MongoDB now would require far too much engineering effort, and for Cool Papers' simple use case, MongoDB's speed wouldn't even match Shelve's anyway.
After much fruitless searching, the author stumbled upon a wonderfully small yet powerful alternative to Whoosh — tantivy, a full-text search library written in Rust that also provides a Python binding, letting it be used as a Python library. Its API is similar to Whoosh's, but it is still being actively maintained. As everyone knows, Rust is famous for its efficiency, so it's fair to say tantivy fulfills every ideal one could imagine for a full-text search library — fast, lightweight, and clean.
Once the full-text search library was chosen, all that remained was the front-end work. In "Happy New Year! Notes on the Development Experience of Cool Papers", the author already confessed to being a complete novice at front-end work with zero artistic sense, so designing the UI was an exceptionally difficult task — all that could be done was endless searching, copy-pasting, and asking GPT-4 and Kimi for help. Through a patchwork of cobbling and tinkering, a usable interface was finally, barely, put together. During development, the original built-in page search was also given an opportunity for optimization along the way, so using the page search should now feel noticeably faster.
Closing Remarks
While everyone was busy reading about KAN (Kolmogorov-Arnold Networks) over the May Day holiday, the author took it easy and, instead of reading papers, spent the time adding a site search feature to Cool Papers. I won't claim it was something "long awaited and finally arrived," but it is a feature some users have been asking about for quite a while, so here's a brief introduction to it, along with a summary of the experience of building it.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.