[Python-ideas] solving multi-core Python

Sturla Molden sturla.molden at gmail.com
Thu Jun 25 17:18:10 CEST 2015

On 25/06/15 16:08, Nick Coghlan wrote:

> It generally isn't scientific programmers I personally hit problems
> with (although we have to allow for the fact many of the scientists I
> know I met *because* they're Pythonistas). For that use case, there's
> not only HPC to point to, but a number of papers that talking about
> Cython and Numba in the same breath as C, C++ and FORTRAN, which is
> pretty spectacular company to be in when it comes to numerical
> computation.

Cython can sometimes give the same performance as C or Fortran, but as 
soon as you start to use classes in the Cython code you run into GIL 
issues. It is not that the GIL is a problem per se, but because Cython 
compiles to C, the GIL is not released until the Cython function 
returns. That is, unless you manually release it inside Cython. This 
e.g. means that the interpreter might be locked for longer durations, 
and if you have a GUI it becomes unresponsive. The GIL is more painful 
in Cython than in Python. Personally I often end up writing a mix of 
Cython and C or C++.

Numba is impressive but still a bit immature. It is an LLVM based JIT 
compiler for CPython that for simple computational tasks can give 
performance similar to C. It can also run Python code on Nvidia GPUs. 
Numba is becoming what the dead swallow should have been.

> Instead, the folks that I think have a more valid complaint are the
> games developers, and the folks trying to use games development as an
> educational tool.

I have not developed games myself, but for computer graphics with OpenGL 
there is certainly no reason to complain. NumPy arrays are great for 
storing vertex and texture data. OpenGL with NumPy is just as fast as 
OpenGL with C arrays. GLSL shaders are just plain text, Python is great 
for that. Cython and Numba are both great if you call glVertex* 
functions the old way, doing this as fast as C. Display lists are also 
equally fast from Python and C. But if you start to call glVertex* 
multiple times from a Python loop, then you're screwed.

> That does suggest to me a possible "archetypal problem" for the work
> Eric is looking to do here: a 2D canvas with multiple interacting
> circles bouncing around. We'd like each circle to have its own
> computational thread, but still be able to deal with the collision
> physics when they run into each other.

There are people doing Monte Carlo simulations with thousands or 
millions of particles, but not with one thread per particle.



More information about the Python-ideas mailing list