[Python-ideas] Optimistic Concurrency

[Adam Olsen]

On Mon, Oct 20, 2008 at 8:11 PM, Dillon Collins <dillonco at comcast.net> wrote:
> On Monday 20 October 2008, Adam Olsen wrote:
>> Stepping back a bit, there's two distinct problems to solve:
>> 1) Making threading easier
>> 2) Removing the GIL
>
> As I've not seen it mentioned in this thread, I feel that it's worth pointing
> out that 2.6's new multiprocessing module allows for programs to use more
> than one CPU, thus eventually eliminating the GIL:
> http://docs.python.org/dev/2.6/library/multiprocessing.html
>
> In light of this I doubt we'll see the GIL removed from CPython pretty much
> ever.  The GIL does have the benefit of speeding up CPython fairly
> significantly by removing lock overhead, and simplifies the code.  It can
> also be released by C libs for things that will take a long time (which are
> often best in C for that reason anyway).
>
> So, while removing the GIL is fun from an intellectual perspective, it's
> practical value is now somewhat limited.  Sure, multiprocessing isn't
> perfect, but it is the first thing to bypass the GIL that's been accepted ;).

The GIL won't be removed from CPython, but that's because it'd be a
near-total-rewrite to do effectively.

multiprocessing is useful when there's a limited amount of shared
data.  Problems that involve large amounts of shared data will still
require threading.  You can gain scalability by moving the logic into
C, but as the number of CPUs increases so does the amount that must be
in C.  Eventually you reach a point where using python is
counterproductive.

As far as I'm concerned we can already solve this: safethread defines
the semantics (#1) and other implementations like PyPy can handle the
scalability (#2).  What we lack is a significant number of people that
*need* this solved.


-- 
Adam Olsen, aka Rhamphoryncus