[Python-Dev] Variant of removing GIL.

Luis P Caamano lcaamano at gmail.com
Sat Sep 17 19:51:29 CEST 2005

On 9/17/05, "Martin v. Löwis" <martin at v.loewis.de> wrote:
> Instead, the issue mainly died because nobody provided
> working code (along with a strategy on what to do with
> the existing extension modules).

When I first started writing python code I had just come out
of about 6 years of kernel development (1994-2000).  These 6 years
span the times when we were making changes to UP kernels
to work correctly and efficiently on SMP machines, implementing
kernel threads, and thread-safing libc and other changes needed
to move from the now defunct user space DCE-threads to posix threads
including kernel threads and even MxN support.  So, I was very
familiar with thread programming.

I architected, designed, and developed a bunch of distributed
servers and services using python and Pyro and it all was done
using nice and cool thread programming practices and techniques.
It was nice to see Python supported threads.

Close to a year into this when we had a lot of functionality implemented
I started running scalability tests and I was shocked and appalled
to find out that our architecture did not scale well on SMP machines
because we depended on threads.  And that's when I discovered
the GIL and twisted and other process based python techniques
used to get around the GIL.  It was too late for us to redo things and
we've been in a holding pattern since then waiting to see what will hit
us first, the need to scale a lot, which means we'd have to drop some
time to implement some changes to support process based scalability,
or no GIL.  I have my money on process based scalability. :(  

I'm sure that has happened to a lot of people because nobody finds
about the GIL in the beginning of their python development
experience.  If I started writing a new complex app from scratch,
now I know exactly what to do.

One big problem is that a lot of people defend the GIL with the
premise that the GIL is OK because we release the GIL in
C extensions and during IO.  That is true, the GIL is not as
bad as it would've been if the CPython interpreter extensions
didn't do that but it's also true that it hurts performance on
SMP machines in more than one way.   When I found out about
this it dawn on me that the performance could've been worse
although it was still way below what we expected.

I don't remember exactly how but what I remember is that when
I presented the problem to Guido he just told me to put up or
shut up.  At first I was insulted but then I realized that it was
a fair response given the complexity of the problem, previous
history I didn't know about, the fairly good compromise of
releasing the GIL in extensions, and the lack of a large
percentage of python developers asking for better performance
on SMP machines.

I couldn't put up so ...  I think I've gone beyond my quota :-)


The GIL is probably my only peeve about the CPython
interpreter.  So don't get me wrong, I love the language and I'm
always grateful for all the hard work you guys put in developing
such great language and implementation.

Luis P Caamano
Atlanta, GA USA

More information about the Python-Dev mailing list