[Python-ideas] Python threads are perfectly fine

[Sturla Molden]

Antoine Pitrou skrev:
> Sturla Molden <sturla at ...> writes:
>   
>> P.S. Yes I know about the GIL. It can be released by C extensions (incl. 
>> Cython/Pyrex, f2py, ctypes). Python threads are perfectly fine for 
>> course-grained parallelism in numerical code, with scalability and 
>> performance close to GCC's OpenMP (I have tried).
>>     
>
> That's a very refreshing thing to hear, thanks :-)
>
>   

I could show you a test I did on my laptop (dual core) a while ago:

http://folk.uio.no/sturlamo/kdtree/benchmark-27022009.png

The black line is scipy.spatial.cKDTree (Cython).
The green line is scipy.spatial.cKDTree modified to use Python threads 
(GIL released whenever possible).
The red line is scipy.spatial.cKDTree with some parts re-written in C, 
and using OpenMP.

This is hardly surprising, as Python threads are just native OS threads. 
The slightly reduced performance of Python threads probably comes from 
contention for the GIL in parts of the Cython code. At least for 
numerical code, the heavy lifiting is done in special C and Fortran 
libraries such as ATLAS/LAPACK, Intel MKL, FFTW, and MINPACK.  Even for 
code we write completely ourselves, there will always be some 
performance critical parts in Cython, C/C++ or Fortran. We can thus 
release the GIL around the worst bottlenecks, and use multi-threading in 
Python. The GIL becomes an issue if you never release it.

S.M.