[Python-ideas] solving multi-core Python
sturla.molden at gmail.com
Thu Jun 25 17:18:10 CEST 2015
On 25/06/15 16:08, Nick Coghlan wrote:
> It generally isn't scientific programmers I personally hit problems
> with (although we have to allow for the fact many of the scientists I
> know I met *because* they're Pythonistas). For that use case, there's
> not only HPC to point to, but a number of papers that talking about
> Cython and Numba in the same breath as C, C++ and FORTRAN, which is
> pretty spectacular company to be in when it comes to numerical
Cython can sometimes give the same performance as C or Fortran, but as
soon as you start to use classes in the Cython code you run into GIL
issues. It is not that the GIL is a problem per se, but because Cython
compiles to C, the GIL is not released until the Cython function
returns. That is, unless you manually release it inside Cython. This
e.g. means that the interpreter might be locked for longer durations,
and if you have a GUI it becomes unresponsive. The GIL is more painful
in Cython than in Python. Personally I often end up writing a mix of
Cython and C or C++.
Numba is impressive but still a bit immature. It is an LLVM based JIT
compiler for CPython that for simple computational tasks can give
performance similar to C. It can also run Python code on Nvidia GPUs.
Numba is becoming what the dead swallow should have been.
> Instead, the folks that I think have a more valid complaint are the
> games developers, and the folks trying to use games development as an
> educational tool.
I have not developed games myself, but for computer graphics with OpenGL
there is certainly no reason to complain. NumPy arrays are great for
storing vertex and texture data. OpenGL with NumPy is just as fast as
OpenGL with C arrays. GLSL shaders are just plain text, Python is great
for that. Cython and Numba are both great if you call glVertex*
functions the old way, doing this as fast as C. Display lists are also
equally fast from Python and C. But if you start to call glVertex*
multiple times from a Python loop, then you're screwed.
> That does suggest to me a possible "archetypal problem" for the work
> Eric is looking to do here: a 2D canvas with multiple interacting
> circles bouncing around. We'd like each circle to have its own
> computational thread, but still be able to deal with the collision
> physics when they run into each other.
There are people doing Monte Carlo simulations with thousands or
millions of particles, but not with one thread per particle.
More information about the Python-ideas