[Python-3000] the future of the GIL

[Guido van Rossum]

On 5/8/07, Luis P Caamano <lcaamano at gmail.com> wrote:
> On 5/7/07, "Guido van Rossum" wrote:
> > Around '99 Greg Stein and Mark Hammond tried to get rid of the GIL.
> > They removed most of the global mutable data structures, added
> > explicit locks to the remaining ones and to individual mutable
> > objects, and actually got the whole thing working. Unfortunately even
> > on the system with the fastest locking primitives (Windows at the
> > time) they measured a 2x slow-down on a single CPU due to all the
> > extra locking operations going on.
>
> That just breaks my heart.<sigh>
>
> You gotta finish that sentence, it was a slow down on single CPU with
> a speed increase with two or more CPUs, leveling out at 4 CPUs or so.
>
> This was the same situation on every major OS kernel, including AIX,
> HPUX, Linux, Tru64, etc., when they started supporting SMP machines,
> which is why all of them at some time sported two kernels, one for SMP
> machines with the spinlock code and one for single processor machines
> with the spinlock code #ifdef'ed out.  For some, like IBM/AIX and
> HPUX, eventually and as expected, all their servers became MPs and
> then they stopped delivering the SP kernel.
>
> The same would've been true for the python interpreter, one for MP and
> one for SP, and eventually, even in the PC world, everything would be
> MP and the SP interpreter would disappear.

The difference is, for an OS kernel, there really isn't any other way
to benefit from multiple CPUs. But for Python, there is -- run
multiple processes instead of threads!

> People need to understand though that the GIL is not as bad as one
> would initially think as most C extensions release the GIL and run
> concurrently on multiple CPUs.  It takes a bit of researching through
> old emails in the python list and a bit of time to really understand
> that.  Nevertheless, when the itch is bad enough, it'll get scratched.

I think you're overestimating the sophistication of the average
extension developer, and the hardware to which they have access.

Nevertheless, you're right the GIL is not as bad as you would
initially think: you just have to undo the brainwashing you got from
Windows and Java proponents who seem to consider threads as the only
way to approach concurrent activities.

Just because Java was once aimed at a set-top box OS that didn't
support multiple address spaces, and just because process creation in
Windows used to be slow as a dog, doesn't mean that multiple processes
(with judicious use of IPC) aren't a much better approach to writing
apps for multi-CPU boxes than threads.

Just Say No to the combined evils of locking, deadlocks, lock
granularity, livelocks, nondeterminism and race conditions.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)