The Distribution of ARXIV Math Papers: Analysis of PDEs Comes Out on Top!

ARXIV数学论文分布：偏微分方程最热门！ 苏剑林 · 2015-11-13 · Machine Learning Python data mining

I've successfully secured a spot in the pure mathematics program at Sun Yat-sen University, a rather theoretical field of study. Even so, I intend to keep up my interest in data analysis, computing, and related areas. These past few days I've been in the mood to do something combining my field with data mining, so I scraped the last five years (2010–2014) of math papers from ARXIV (collecting title, category, year, and month for each), hoping to run a simple analysis of the math "market" over these years. Personally, I think that since ARXIV is currently the world's largest electronic database of preprints, analyzing its data should yield conclusions with a certain degree of representativeness.

Of course, this post is really just an exercise in web scraping and basic data analysis — it doesn't dig up any particularly valuable insights. At the end of the post I've attached the data I scraped, for readers who are interested in doing further analysis.

Overall Picture

Over these five years, the total number of math papers on ARXIV was 135,009, averaging about 27,000 per year, or 74 per day. more

Looking at categories, the top fifteen by number of papers are:

This table, in a sense, reflects how popular each direction in mathematics currently is. Ranked first is analysis of PDEs, which is somewhat related to mathematical physics in fourth place — both largely represent applications of partial differential equations, especially in physics, biology, and other fields. Second is probability theory; since almost every phenomenon in our world carries some element of randomness, it's only natural that this direction has flourished, so its popularity makes perfect sense. Third is combinatorics, representative of discrete mathematics. Fifth is information theory, which is likely a result of the recent boom in data mining. Further down we have algebraic geometry, number theory, differential geometry, dynamical systems, functional analysis, numerical analysis, optimization and control — these should all be relatively cutting-edge and popular areas within mathematics.

Next, I split up the titles to see which words appear most often. As one would expect, the most frequent are meaningless stop words like of, and, the, for, in, with, a, on. After removing these stop words, the result is:

equations(5172), groups(4782), spaces(4531), systems(4422), random(3980), functions(3906), quantum(3817), equation(3720), algebras(3686), theory(3459), graphs(3437), problem(3337), finite(3275), model(3216), solutions(3097), theorem(3014), operators(2880), linear(2718), generalized(2622), type(2579), group(2565), space(2402), manifolds(2363), analysis(2315), stochastic(2278), problems(2235), models(2161), surfaces(2156), applications(2060), nonlinear(2017), approach(1961), local(1930), polynomials(1922), method(1919), fields(1886), differential(1882), new(1874), optimal(1869), function(1854), boundary(1789), number(1768), sets(1766), curves(1751)

The first word, equations, and the eighth, equation, presumably correspond to the "analysis of PDEs" category, followed by groups (group theory), spaces, and so on — likely reflecting the mainstream approach in current mathematical research, namely placing objects of study within some space and studying them using functional analysis combined with abstract algebra (especially group theory). Interestingly, the word quantum also ranks quite high, suggesting that mathematical research set against the backdrop of quantum theory is also flourishing. I'll leave the rest for readers to judge for themselves.

Year-by-Year Changes

Having covered the overall picture, we can now look at year-by-year changes. First, the total number of papers each year, which keeps increasing every year:

Total papers per year

Next, let's look at the five categories with the most papers over these five years, to see which fields are gradually becoming more popular.

2010 Mathematical Physics(1619) Probability(1437) Algebraic Geometry(1358) Analysis of PDEs(1319) Combinatorics(1297)
2011 Mathematical Physics(1809) Probability(1671) Combinatorics(1605) Analysis of PDEs(1545) Algebraic Geometry(1414)
2012 Mathematical Physics(2005) Analysis of PDEs(1319) Combinatorics(1826) Probability(1824) Information Theory(1616)
2013 Analysis of PDEs(2211) Probability(2027) Combinatorics(2020) Information Theory(1958) Mathematical Physics(1773)
2014 Analysis of PDEs(2464) Combinatorics(2189) Probability(2105) Information Theory(2008) Mathematical Physics(1646)

We can see that mathematical physics steadily held first place for the first three years, but in the last two years, even as the total number of papers increased, the number of mathematical physics papers dropped fairly substantially — this seems to suggest that the field might be hitting some kind of bottleneck? On the other hand, the category that has been climbing year over year and has gradually risen to first place is analysis of PDEs, which suggests that research on systems of partial differential equations has remained a mainstream area of contemporary mathematical research all along.

Let's see which categories are growing the fastest. Below I've picked out a few that I think are fairly representative.

The first is Systems and Control (cs.SY): the number of papers over these five years was, in order, 9, 96, 112, 139, 135. Somewhat related to this is Optimization and Control (math.OC), whose paper counts over the five years were 423, 545, 778, 980, 1289.

Systems and control

Optimization and control

In addition, numerical analysis has also become increasingly popular, with paper counts rising year after year at a fairly large rate — the five-year counts were 435, 571, 778, 1012, 1262. These trends indicate that the combination of mathematics and computing is one of the mainstream directions in the development of mathematics. Other categories reflecting this trend include Computational Physics (physics.comp-ph), Computational Geometry (cs.CG), and Computer Vision and Pattern Recognition (cs.CV).

Numerical analysis

Related fields

I used a simple metric to measure the growth rate of a given category:

$$\sum_{n=2010}^{2013}\frac{(n+1)\text{papers per year}}{n\text{papers per year}}$$

Let me state up front that this metric is very simple and not necessarily accurate — it's meant purely as an intuitive gauge. The categories with the fastest growth rate as identified by this metric are listed in the table below. Interestingly, quite a few of these fields have some connection to computer science, and I don't think that's a coincidence.

Attachment Download

Finally, here's the file I scraped, for readers who are interested in doing further analysis.

arxiv.zip