[Python-ideas] Python threads are perfectly fine
sturla at molden.no
Tue Oct 13 17:13:49 CEST 2009
Antoine Pitrou skrev:
> Sturla Molden <sturla at ...> writes:
>> P.S. Yes I know about the GIL. It can be released by C extensions (incl.
>> Cython/Pyrex, f2py, ctypes). Python threads are perfectly fine for
>> course-grained parallelism in numerical code, with scalability and
>> performance close to GCC's OpenMP (I have tried).
> That's a very refreshing thing to hear, thanks :-)
I could show you a test I did on my laptop (dual core) a while ago:
The black line is scipy.spatial.cKDTree (Cython).
The green line is scipy.spatial.cKDTree modified to use Python threads
(GIL released whenever possible).
The red line is scipy.spatial.cKDTree with some parts re-written in C,
and using OpenMP.
This is hardly surprising, as Python threads are just native OS threads.
The slightly reduced performance of Python threads probably comes from
contention for the GIL in parts of the Cython code. At least for
numerical code, the heavy lifiting is done in special C and Fortran
libraries such as ATLAS/LAPACK, Intel MKL, FFTW, and MINPACK. Even for
code we write completely ourselves, there will always be some
performance critical parts in Cython, C/C++ or Fortran. We can thus
release the GIL around the worst bottlenecks, and use multi-threading in
Python. The GIL becomes an issue if you never release it.
More information about the Python-ideas