[Python-Dev] Fwd: Removal of GIL through refcounting removal.

[Michael Foord]

Calvin Spealman wrote:
> Has anyone made the argument for keeping the GIL to discourage
> threading? I'm only throwing this out there and I'm sure we'd want to
> improve things no matter what, but I would like to voice the concern
> anyway. We all know there are people who think threading is the answer
> to all things, and who don't understand the difficulties and problems
> with threading. Often, when trying to persuade them of alternative
> solutions, one selling point is "your python threads can't take
> advantage of multiple cores, anyway, because of the GIL."
>
>   

There are many concurrency problems for which multiprocessing is *not* a 
good answer and for which careful use of threads is a relatively 
straightforward answer. For people with those sorts of problems CPython 
(and therefore the perception is Python) does not offer a usable 
concurrency story unless you are prepared to also write in C.

Unfortunately many in the Python community seem pathologically opposed 
to even good use of threading and refuse to believe that such situations 
even exist. In the meantime, people with these problems (or who believe 
they are likely to have such problems) simply turn to other languages.

All the best,

Michael Foord

> I worry, if it is ever actually removed, that I won't be able to
> convince these people that you really don't need to spawn twenty
> threads to write an IRC bot.
>
> On Fri, Oct 31, 2008 at 2:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>   
>> Neil Schemenauer wrote:
>>     
>>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>>       
>>>> For a student project in a course on virtual machines, we are
>>>> evaluating the possibility to experiment with removing the GIL
>>>> from CPython
>>>>         
>>> Hi,
>>>
>>> It's great to hear of this kind of project.  I think what you want
>>> to do is difficult but possible.  The major compilcation would be
>>> that extension modules would have to re-written since they all
>>> assume a reference counting GC.  A foreign function interface like
>>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>>> does place different demands on extension module authors.
>>>       
>> Michael Foord's comment about the way Ironclad does it suggests a
>> possible interim step that would allow existing extensions to be used at
>> the cost of some additional overhead: use free threading in the core,
>> but have an "extension module lock" that cuts things back to a single
>> thread whenever non-core code is invoked.
>>
>> It wouldn't be as fast as a completely free-threaded setup (since the
>> EML would need to operate as a variant of a 'read/write' lock, with the
>> core code mostly leaving it in the multiple-threads-allowed 'read' mode,
>> promoting it to single-thread-only 'write' mode when potentially
>> dispatching to non-core code, and dropping back to the 'read' mode when
>> the dispatch landed back in core code again).
>>
>> Over time, extension modules which were updated to use their own
>> internal locks could also drop back to free-threading mode. In essence,
>> the GIL would still exist, but instead of being a simple mutex the way
>> it is now, it would have an intermediate stage which allowed multiple
>> threads to continue running. Once most extension modules had been
>> converted, the function to promote the EML from multi-threaded mode to
>> single-thread only mode could be removed.
>>
>> A mechanism like that would avoid one of the more subtle problems in
>> removing the GIL: currently, CPython-specific Python code can assume
>> that invocation of a C function from Python code is an atomic operation,
>> since the body of the function executes as a single bytecode (we're
>> talking about the case here where the C function is known not to release
>> the GIL).
>>
>> I've written code which relied on the GIL in exactly that way, and I'm
>> sure I'm far from the only Python programmer to do so. If the GIL was
>> removed without adding an extension module lock, such applications would
>> require additional locking either in the C extension module or in the
>> Python code.
>>
>> A possible starting point for such a project: move the locking
>> mechanisms that are implemented in Python in the threading module to
>> C-based implementations in _thread so that a richer low overhead locking
>> arrangement is available to extensions modules (and the rest of the core).
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> ---------------------------------------------------------------
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>>
>>     
>
>
>
>   


-- 
http://www.ironpythoninaction.com/