[Baypiggies] Guido's Blog: It isn't Easy to Remove the GIL
Shannon -jj Behrens
jjinux at gmail.com
Fri Sep 14 00:09:41 CEST 2007
On 9/11/07, Stephen McInerney <spmcinerney at hotmail.com> wrote:
> [A great discussion of something complex which I was too timid to ask about
> on the list:]
> Article: http://www.artima.com/weblogs/viewpost.jsp?thread=214235
> Responses: http://www.artima.com/forums/flat.jsp?forum=106&thread=214235
> - Essentially the GIL will not go away and the C-implementation of Python
> will stay single-threaded, unless some code wizard _other than Guido_ goes
> through the effort of
> removing it (e.g. for 3.0), and showing that its removal doesn't slow down
> single-threaded Python code.
> - Includes description of the unpromising prior attempt at this on a fork of
> 1.5 with 2x slowdown
> Guido> "[...] I'd welcome it if someone did another experiment along the
> lines of Greg's patch (which I haven't found online), and I'd welcome a set
> of patches into Py3k only if the performance for a single-threaded program
> (and for a multi-threaded but I/O-bound program) does not decrease.
> I would also be happy if someone volunteered to maintain a GIL-free fork of
> Python, in case that the single-threaded performance goal can't be met but
> there is significant value for multi-threaded CPU-bound applications. We
> might even end up with all the changes permanently part of the code base,
> but enabled only on request at compile time.
> However, I want to warn that there are many downsides to removing the GIL.
> It complicates life for extension modules, who can no longer expect that
> they are invoked in a "safe zone" protected by the GIL -- as soon as an
> extension has any global mutable data, will have to be prepared with
> concurrent calls from multiple threads. There might also be changes in the
> Python/C API necessitated by the need to lock certain objects for the
> duration of a sequence of calls.
> While it is my personal opinion, based upon the above considerations, that
> there isn't enough value in removing the GIL to warrant the effort, I will
> welcome and support attempts to show that times have changed. However, there
> is no point in pleading alone -- Python is open source and I have my hands
> full dealing with the efforts to produce a quality 3.0 language definition
> and implementation on time. I want to point out one more time that the
> language doesn't require the GIL -- it's only the CPython virtual machine
> that has historically been unable to shed it."
<jj crazy mode>
For several months, I've been fascinated by the idea of a Pythonic
version of Erlang. The most important thing is to combine
Erlang-style concurrency with Python's elegant syntax.
However, there are some real challenges here. In Python, classes,
instances, modules, etc. are dicts, which is a fundamental problem
when trying to support Erlang-style concurrency. Message passing
(within process) is cheap in Erlang because they can pass by reference
instead of by value and rest assured that the data will not be
modified. Remember, Erlang gets away from the need for locks because
two "processes" cannot modify the same data. That's a real challenge
for a language like Python where even writing a def is adding a new
value to a dict (for the class or module). Sure, you can make it so
that you can only pass tuples of strings and ints, but that's no fun
because what you probably want to pass is some combination of dicts,
lists, and instances.
In Haskell, there's a tree data type that is optimized for
non-destructive updates. Just like you can create a new list by
consing an element with another list without harming the other list,
you can do the same thing with a tree. Of course, it's a little less
efficient than "updating" a list. I wonder if it'd be possible to
replace Python's reliance on dicts with such a tree. Hence, adding a
new def would update your own processes view of the tree without
harming the old tree which all the other processes are pointing to.
There are other difficulties, but if I actually had two years of free
time and about 20 extra IQ points, and money to live on, I'd probably
take a shot at it ;)
</jj crazy mode>
More information about the Baypiggies