Using PyPy to Improve Python Script Execution Efficiency

用PyPy提高Python脚本执行效率 苏剑林 · 2014-06-11 · Life-Feeling prime number Python

In The Sum of Primes Below Two Million and the Sum of the First Two Million Primes, we used Python to compute the sum of the first two million primes and the sum of primes below two million, and got the following execution times in Python 3.3:

Sum of primes below two million:
142913828922
time: 2.4048174478605646
Sum of the first two million primes:
31381137530481
time: 46.75734807838953

So I started looking for ways to improve the execution efficiency of the Python script. I felt there wasn't much room left for improvement on the algorithmic side, so I turned to optimizing the interpreter instead. While searching around, I stumbled upon a term — Psyco! This is an external module for Python that, once imported, speeds up the execution of .py scripts. There are also articles online such as "Using Psyco to Make Python Run as Fast as C" and "Using psyco to Speed Up Python Program Execution," which suggest that Psyco is indeed a viable option. So I was eager to give it a try — but then I found out that Psyco stopped being developed back in 2012 and only supports up to Python 2.4. It has since been succeeded by PyPy. So I downloaded PyPy. more

PyPy is no longer an external module for Python but a standalone interpreter — meaning we can execute a .py script with the command pypy prime.py. It's currently based on Python 2.7.6, and a Windows version is available. How well does it perform? Take a look:

Sum of primes below two million:
142913828922
('time:', 0.226615647130835)
Sum of the first two million primes:
31381137530481
('time:', 9.033084048872064)

One case went from 2.4 seconds down to 0.3 seconds, and the other from 47 seconds down to 9 seconds — a speedup of several times! Running these two scripts with PyPy was indeed satisfying.

So does that mean every Python script can be sped up simply by switching to PyPy? Not necessarily. There are quite a few cases where plain Python actually runs faster — for instance, the script below for computing $10000!$ (only computing, without any output):

import time
start=time.clock()
s=1
for i in range(1,10000+1):
    s=s*i

end=time.clock()
print(end-start)

In Python 3.3, this takes only 0.06 seconds, but in PyPy it takes 0.3 seconds. And when computing $100000!$, PyPy takes 39 seconds, while Python 3.3 only needs 7 seconds!

Regarding PyPy versus Python, here's the best answer from Veedrac:

As others have mentioned, PyPy has quite weak support for C extensions. It does support C extensions, but they end up running slower than pure Python itself. As a result, a lot of modules still require CPython.
Numpy on CPython has excellent data-processing performance, satisfying people who need both speed and the heavy use of data-analysis libraries like Pandas and SciPy.
So PyPy either lacks proper C extension support or has very weak support for it — and even when it does support it, it slows down that kind of data processing. It simply can't compete with the combination of speed and ease of use that CPython offers.
Second, Python 3 support in PyPy is still experimental at this stage. People who rely on the newest features of the latest Python versions probably aren't willing to give up those shiny new features yet.
Third, PyPy isn't actually fast for scripts, and most people using Python are, in fact, writing scripts — short programs. PyPy's biggest strength is its just-in-time (JIT) compiler, which is designed for long-running, simple numerical computations. To put it plainly, PyPy's upfront compilation overhead is much larger than CPython's.
Fourth, inertia. Switching to PyPy requires re-provisioning machines, which is simply too much extra work for many users or organizations.

That said, for me personally, most of the programs I write these days are scientific computing scripts with a lot of repeated computation, and PyPy really does improve efficiency significantly! So going forward, I'll probably try to use PyPy for execution as much as possible.