[Python-Dev] Fwd: Removal of GIL through refcounting removal.

[Adam Olsen]

On Fri, Oct 31, 2008 at 2:11 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Adam Olsen wrote:
>> On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Neil Schemenauer wrote:
>>>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>>>> For a student project in a course on virtual machines, we are
>>>>> evaluating the possibility to experiment with removing the GIL
>>>>> from CPython
>>>> Hi,
>>>>
>>>> It's great to hear of this kind of project.  I think what you want
>>>> to do is difficult but possible.  The major compilcation would be
>>>> that extension modules would have to re-written since they all
>>>> assume a reference counting GC.  A foreign function interface like
>>>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>>>> does place different demands on extension module authors.
>>> Michael Foord's comment about the way Ironclad does it suggests a
>>> possible interim step that would allow existing extensions to be used at
>>> the cost of some additional overhead: use free threading in the core,
>>> but have an "extension module lock" that cuts things back to a single
>>> thread whenever non-core code is invoked.
>>
>> Ahh, or do it per object!  Convert the core to use a precise GC
>> references, but retain the refcount API as a sort of forced unknown
>> reference.  Needs bodging to retain our current handling of cycles in
>> refcounted objects, but it should be doable, and it potentially has a
>> lot less overhead (and is more scalable) than what I've got now with
>> safethread.
>
> The GIL protects more than just refcounting: it is used widely to
> protect access to shared data structures both inside and outside the
> core. Until all the extension modules implement their own locks instead
> of relying on the GIL, some form of global lock needs to remain if you
> want to preserve compatibility with existing extensions. And since being
> able to use existing extensions is the main advantage of using CPython
> rather than just writing a separate free-threaded Python VM, that seems
> like a fairly important design goal.

safethread already handles that, via monitors.  The main thread is
given a monitor that all old, unshareable extension modules exist in.
Once they're updated and marked as shareable they can be used from any
monitor in any thread.


-- 
Adam Olsen, aka Rhamphoryncus