[Python-Dev] GIL musings (was Re: Thoughts fresh after EuroPython)

[Ronald Oussoren]

On 28 Jul, 2010,at 12:56 PM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:

On 28/07/2010 11:50, Nick Coghlan wrote:
> On Tue, Jul 27, 2010 at 12:33 AM, Ronald Oussoren
> <ronaldoussoren at mac.com> wrote:
> 
>> In my opinion the GIL is a weak point of CPython and it would be nice if it
>> could be fixed. That is however easier said than done, a number of people
>> have tried in the past and ran into implementation limitations like our
>> refcounting garbage collector that make hard to remove the GIL without
>> either rewriting lots of code, or running into a brick wall
>> performance-wise.
>>
>> The HotPy presentation at EuroPython shows that it is possible to remove the
>> GIL, although at the cost of replacing the garbage collector and most likely
>> breaking existing C extensions (although the HotPy author seemed to have a
>> possible workaround for that).
>> 
> This is the kind of approach that seems to hold the most promise of
> removing the GIL without incurring the single-threaded performance hit
> that has been the achilles heel of previous attempts at creating a
> free-threaded CPython implementation. With first IronClad and now PyPy
> blazing the trail in interfacing a garbage collected Python
> implementation with deterministic refcounting for C extension modules,
> it seems plausible that this kind of approach may eventually prove
> acceptable.
>
> Furthermore, the with statement now provides a superior alternative to
> application level tricks that previously relied on deterministic
> refcounting.
>
> While multi-threading does break down beyond a certain number of
> cores, it *is* possible to do safely (particularly using queues to
> pass references around) and can avoid plenty of serialisation overhead
> when dealing with sizable data structures.
> 

Breaking binary compatibility with C extensions would be "difficult" 
once PEP 384 (stable binary ABI) has gone into effect. As you intimate, 
Ironclad demonstrates that C extensions *can* be interfaced with a 
different garbage collection system whilst maintaining binary 
compatibility. It does impose constraints however (which is why the PyPy 
c-ext implementors chose source compatibility rather than binary 
compatibility).
 
The HotPy author mentioned that he has a scheme where refcounts could be used by C extensions while the system natively uses a copying collector, but I got the impression that this was not fully fleshed out yet.

Apple's Objective-C garbage collector has a simular feature: you can use CFRetain/CFRelease to manage refcounts and the GC will only collect objects where the CF reference count is 0.  This is a non-copying collector in a C environment though, which makes this scheme easier to implement than with a full generational copying collector.

It should therefore be possible to have an interpreter where the VM uses a real GC and while extensions using the stable ABI could work as is, but that probably requires that Py_INCREF and Py_DECREF expand into function calls in the stable ABI.

Implementing this would still be a significant amount of work.

Ronald



Michael



> Cheers,
> Nick.
>
> 


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100728/407ee1b6/attachment.html>