[Python-Dev] Fwd: Removal of GIL through refcounting removal.

[Calvin Spealman]

Has anyone made the argument for keeping the GIL to discourage
threading? I'm only throwing this out there and I'm sure we'd want to
improve things no matter what, but I would like to voice the concern
anyway. We all know there are people who think threading is the answer
to all things, and who don't understand the difficulties and problems
with threading. Often, when trying to persuade them of alternative
solutions, one selling point is "your python threads can't take
advantage of multiple cores, anyway, because of the GIL."

I worry, if it is ever actually removed, that I won't be able to
convince these people that you really don't need to spawn twenty
threads to write an IRC bot.

On Fri, Oct 31, 2008 at 2:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Neil Schemenauer wrote:
>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>> For a student project in a course on virtual machines, we are
>>> evaluating the possibility to experiment with removing the GIL
>>> from CPython
>>
>> Hi,
>>
>> It's great to hear of this kind of project.  I think what you want
>> to do is difficult but possible.  The major compilcation would be
>> that extension modules would have to re-written since they all
>> assume a reference counting GC.  A foreign function interface like
>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>> does place different demands on extension module authors.
>
> Michael Foord's comment about the way Ironclad does it suggests a
> possible interim step that would allow existing extensions to be used at
> the cost of some additional overhead: use free threading in the core,
> but have an "extension module lock" that cuts things back to a single
> thread whenever non-core code is invoked.
>
> It wouldn't be as fast as a completely free-threaded setup (since the
> EML would need to operate as a variant of a 'read/write' lock, with the
> core code mostly leaving it in the multiple-threads-allowed 'read' mode,
> promoting it to single-thread-only 'write' mode when potentially
> dispatching to non-core code, and dropping back to the 'read' mode when
> the dispatch landed back in core code again).
>
> Over time, extension modules which were updated to use their own
> internal locks could also drop back to free-threading mode. In essence,
> the GIL would still exist, but instead of being a simple mutex the way
> it is now, it would have an intermediate stage which allowed multiple
> threads to continue running. Once most extension modules had been
> converted, the function to promote the EML from multi-threaded mode to
> single-thread only mode could be removed.
>
> A mechanism like that would avoid one of the more subtle problems in
> removing the GIL: currently, CPython-specific Python code can assume
> that invocation of a C function from Python code is an atomic operation,
> since the body of the function executes as a single bytecode (we're
> talking about the case here where the C function is known not to release
> the GIL).
>
> I've written code which relied on the GIL in exactly that way, and I'm
> sure I'm far from the only Python programmer to do so. If the GIL was
> removed without adding an extension module lock, such applications would
> require additional locking either in the C extension module or in the
> Python code.
>
> A possible starting point for such a project: move the locking
> mechanisms that are implemented in Python in the threading module to
> C-based implementations in _thread so that a richer low overhead locking
> arrangement is available to extensions modules (and the rest of the core).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy