On 21/12/14 02:15, Nick Coghlan wrote:
There's also the fact that with both Numba and PyPy now supporting selective JIT acceleration of decorated functions within the context of a larger CPython application, as well as Cython's existing support for precompilation as a C extension, the pattern of profiling to find performance critical areas, and finding ways to optimise those, now seems well established.
Numba is not very mature, but it can already JIT accelerate most of Python bytecode to speeds comparable to -O2 in C. And nit just that, it does not need type-hints, it has also an "autojit" that can infere types at runtime. What the Unladen Swallow failed and Numba succeded? I think for two reasons: First, Numba was designed for a particular purpose: numerical computing. No offence to Google, but the numerics folks are the ones who really understand how to beat gigaflops out of the CPU. Numba was created by Travis Oliphant, who was also the creator of the modern NumPy package (as opposed to Jim Hugunin's original Numerics package and NASA's Numarray packages). The Unladen Swallow did the mistake of trying to accelerate "everything". Second, LLVM has improved. At the time the swallow was hatched, LLVM was not really useful as a JIT compiler framework. It was in comparison to GCC even a lousy static compiler. Now it has matured and it is excellent as as a JIT compiler as well as a static compiler. It is still a little behind GCC and Intel compilers on static optimization, but more than good enough to be the default compiler on MacOSX and FreeBSD. It should also be mentioned that the Numba (or just NumbaPro?) integrates with CUDA, and it can actually run Python bytecode on the GPU. For certain data-parallel tasks it can make your Python program yield a teraflop with a modern Nvidia GPU. I also want to mention Cython. It has also shown that compiled Python can often run at the speed of C++. But its real merit is the ease with which we can write C extensions for Python. I think in the long run, CPython could benefit from Numba as well as Cython. Numba can be used to boost performance critical code without having to resort to C. Cython can e.g. be used to implement the standard library without hand-written C modules. It would lower the threshold for contribution as well as improve readability. Sturla