[Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

[John J Lee]

On Fri, 16 Sep 2005, "Martin v. Löwis" wrote:

> Sokolov Yura wrote:
> > I think I know how to remove GIL!!!! Obviously I am an idiot.
> 
> Not an idiot, just lazy :-) Please try to implement your ideas,
> and I predict that you will find:
> 1. it is a lot of work to implement
> 2. it requires changes to all C files, in particular to extension
>    modules outside the Python source tree proper.
> 3. performing the conversion, even in a semi-mechanical way, will
>    introduce many new bugs, in the form of race conditions because
>    of missing locks.
> 
> Optionally, you may also find that the performance of the
> interpreter will decrease.
[...]

Given the points you make, and the facts that both Python 3 and real
problems with continuing to scale down semiconductor chip feature sizes
are on the horizon, it seems that now would be an excellent time to start
work on this, with the goal of introducing it at the same time as Python
3.

a. Python 3.0 will break lots of code anyway, so the extension module 
   issue becomes far less significant.

b. In x years time (x < 10?) it seems likely multiprocessor (MP) users 
   will be in the majority.  (As a result, the uniprocessor (UP) slowdown 
   becomes less important in practice, and also Python has the opportunity 
   of avoiding the risk of being sidelined by a real or perceived lack of
   MP performance.)

c. Since time is needed to iron out bugs (and perhaps also to reimplememt 
   some pieces of code "from scratch"), very early in the life of Python 3 
   seems like the least-worst time to begin work on such a change.

I realize that not all algorithms (nor all computational problems) scale
well to MP hardware.  Is it feasible to usefully compile both MP and a UP
binaries from one Python source code base?

(I'm also quite aware that the GIL does not prevent all means of achieving
efficient use of multiprocessors.  I'm just concious that different
parellisation problems are presumably best expressed using different
tools, and that Python 3 and increased prevalance of MP systems might tip
the balance in favour of removing the GIL.)

Of course, it still takes a (anti-)hero to step forward and do the work...


John