[Python-Dev] Fwd: Removal of GIL through refcounting removal.

Nick Coghlan ncoghlan at gmail.com
Fri Oct 31 07:24:57 CET 2008

Neil Schemenauer wrote:
> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>> For a student project in a course on virtual machines, we are
>> evaluating the possibility to experiment with removing the GIL
>> from CPython
> Hi,
> It's great to hear of this kind of project.  I think what you want
> to do is difficult but possible.  The major compilcation would be
> that extension modules would have to re-written since they all
> assume a reference counting GC.  A foreign function interface like
> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
> does place different demands on extension module authors. 

Michael Foord's comment about the way Ironclad does it suggests a
possible interim step that would allow existing extensions to be used at
the cost of some additional overhead: use free threading in the core,
but have an "extension module lock" that cuts things back to a single
thread whenever non-core code is invoked.

It wouldn't be as fast as a completely free-threaded setup (since the
EML would need to operate as a variant of a 'read/write' lock, with the
core code mostly leaving it in the multiple-threads-allowed 'read' mode,
promoting it to single-thread-only 'write' mode when potentially
dispatching to non-core code, and dropping back to the 'read' mode when
the dispatch landed back in core code again).

Over time, extension modules which were updated to use their own
internal locks could also drop back to free-threading mode. In essence,
the GIL would still exist, but instead of being a simple mutex the way
it is now, it would have an intermediate stage which allowed multiple
threads to continue running. Once most extension modules had been
converted, the function to promote the EML from multi-threaded mode to
single-thread only mode could be removed.

A mechanism like that would avoid one of the more subtle problems in
removing the GIL: currently, CPython-specific Python code can assume
that invocation of a C function from Python code is an atomic operation,
since the body of the function executes as a single bytecode (we're
talking about the case here where the C function is known not to release
the GIL).

I've written code which relied on the GIL in exactly that way, and I'm
sure I'm far from the only Python programmer to do so. If the GIL was
removed without adding an extension module lock, such applications would
require additional locking either in the C extension module or in the
Python code.

A possible starting point for such a project: move the locking
mechanisms that are implemented in Python in the threading module to
C-based implementations in _thread so that a richer low overhead locking
arrangement is available to extensions modules (and the rest of the core).


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list