[Python-ideas] A concurrency survey of sorts

[Stefan Behnel]

Adam Jorgensen, 04.11.2011 08:53:
> On 4 November 2011 09:41, Nick Coghlan wrote:
>> On Fri, Nov 4, 2011 at 5:03 PM, Adam Jorgensen wrote:
>>> The GIL makes them pseudo-pointless in CPython anyway and the headaches
>>> arising from threading are very frustrating.
>>
>> This is just plain false.

The first part, yes. The second - depends. Threading, especially when 
applied to the wrong task, is a very good way to give you headaches.


>> Threads are still an excellent way to take a
>> synchronous operation and make it asynchronous. Take a look at
>> concurrent.futures in 3.2, which makes it trivial to take independent
>> blocking tasks and run them in parallel. The *only* time the GIL
>> causes problems is when you have CPU bound threads written in pure
>> Python. That's only a fraction of all of the Python apps out there,
>> many of which are either calling out to calculations in C or FORTRAN
>> (scientific community, financial community) or else performing IO
>> bound tasks (most everyone else with a network connection).
>
> I would love to seem some actual stats on this? How many multi-threaded
> apps are hitting the GIL barrier, etc...

In the numerics corner, multi-threaded CPU bound code is surprisingly 
common. In multi-server setups, people commonly employ MPI&friends, but 
especially on multi-core machines (usually less than 64 cores), threading 
is quite widely used. Proof? Cython just gained support for OpenMP based 
parallel loops, due to popular request.

However, computational code usually doesn't hit the "GIL barrier", as you 
call it, because the really heavy computations don't run in the interpreter 
but straight on the iron. So, no, neither I/O bound tasks nor CPU bound 
numerics tasks get in conflict with the GIL, unless you do something wrong.

That's the main theme, BTW. If you're a frequent reader of c.l.py, you'll 
quickly notice that those who complain most loudly about the GIL usually 
just do so because they do something wrong. Threading isn't the silver 
bullet that you shoot at the task at hand. It's *one* way to solve *some* 
kinds of concurrency problems, and certainly not a trivial one.

Stefan