[Python-Dev] Fwd: Removal of GIL through refcounting removal.

Jesse Noller jnoller at gmail.com
Fri Oct 31 15:03:57 CET 2008


On Fri, Oct 31, 2008 at 9:17 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Calvin Spealman wrote:
>>
>> Has anyone made the argument for keeping the GIL to discourage
>> threading? I'm only throwing this out there and I'm sure we'd want to
>> improve things no matter what, but I would like to voice the concern
>> anyway. We all know there are people who think threading is the answer
>> to all things, and who don't understand the difficulties and problems
>> with threading. Often, when trying to persuade them of alternative
>> solutions, one selling point is "your python threads can't take
>> advantage of multiple cores, anyway, because of the GIL."
>>
>>
>
> There are many concurrency problems for which multiprocessing is *not* a
> good answer and for which careful use of threads is a relatively
> straightforward answer. For people with those sorts of problems CPython (and
> therefore the perception is Python) does not offer a usable concurrency
> story unless you are prepared to also write in C.
>
> Unfortunately many in the Python community seem pathologically opposed to
> even good use of threading and refuse to believe that such situations even
> exist. In the meantime, people with these problems (or who believe they are
> likely to have such problems) simply turn to other languages.
>
> All the best,
>
> Michael Foord

There's room in python for both (all).

Threading, as given to us by Java and others is hard to get right and
is used by many because its "just there", but it is *still* useful for
many people. If python were to have free threading, courtesy of a lack
of the GIL, it would help those people quite a bit. Sometimes you just
need shared state. Myself? I used multiprocess *and* threads all the
time for various reasons.

Of course: Does shared state make more sense as implemented via the
Erlang green threads system - wherein threads are independent of one
another, but pass messages between one another via the parent? (Van
and I were talking about this for some time yesterday :))

But fundamentally - there's still an attraction to things like
IPC/Multiprocessing, Actors, message passing/etc as well. I don't see
free threading and other language improvements for additional styles
of concurrent/distributed programming as mutually exclusive.

Excluding the possibility of free-threading because "we don't like it"
is sorta silly. It would be equivalent of blocking aynsc libraries for
the same reason.

All of that being said: I think there's room to aim for a *better*
threading model, or even a series of higher level abstractions (on top
of the basic abstractions threading provides on top of thread for
example). For example: if you were building a Java application today,
would you use the basic java.lang.thread classes and all of the other
locking primitives, or would you go and use the abstractions and
utilities in the java.util.concurrent package?

Most people would reply with the latter: The language now provides to
you a proven set of tools which have been "proven" to work, are "safe"
- it doesn't make sense for them to go and build out their own thread
factory and what not.

Imagine a concurrent package for python: in it you have your basics -
threading, multiprocessing (maybe "acting" too, but that name sucks,
how about greenprocesses? :)) and you also have utilities like
multiprocessing.pool, multiprocessing.manager/etc,
threading.producer_consumer, etc. We build these out and make them "as
safe as possible".

Over time, we begin to hide more and more of the primitives and
encourage usage of safer, proven abstractions. Take a look at
something like Kamaelia - it's a dog pile of abstractors and metaphors
that make it dead simple to do concurrent things without exposing the
end-programmer to a lot of the underlying pain and basics. Sure: If
I'm and end user I have to decide "threads, processes or green
processes" but that's really an extension of the problem domain.
Threads aren't good for some problems, processes are bad for others,
etc.

Just some pie-in-the-sky thoughts. Fundamentally, the GIL is not an
end-user "feature" unless that end user is writing extension modules -
and in that case, it's a pretty nice feature. However end users
*benefit* from the GIL by having a rich standard library and a much
more rich ecosystem of 3rd party modules - and that benefit can't be
understated or sacrificed.

-jesse


More information about the Python-Dev mailing list