GIL in the new glossary

[Tim Peters]

[Luis P Caamano]
> The glossary at <http://www.python.org/dev/doc/devel/tut/node16.html>
> has the following definition for the GIL:

>> global interpreter lock
>>
>> The lock used by Python threads to assure that only one thread
>> can be run at a time. This simplifies Python by assuring that no
>> two processes can access the same memory at the same time. Locking
>> the entire interpreter makes it easier for the interpreter to be
>> multi-threaded, at the expense of some parallelism on
>> multi-processor machines.

> Some parallelism???  Wouldn't it be more accurate to say
> "at the expense of parallelism?"

In the most practical sense, no, it wouldn't.  The most important kind of
parallelism for most Python apps is getting multiple I/O channels working
"in parallel", and Python's C code releases the GIL around almost all calls
to C I/O functions.  This allows other Python threads to make progress while
the I/O operation completes (whether it's a disk write or a socket
operation).

> The GIL doesn't eliminate "some" paralellism, it completely
> eliminates any chance of parallelism within the same interpreter.

It's subtler than that, of course.  C code is free to release the GIL when
appropriate, and lots of the C code does.

>> Efforts have been made in the past to create a "free-threaded"
>> interpreter (one which locks shared data at a much finer
>> granularity), but performance suffered in the common
>> single-processor case.

> This is true for kernels too, which is why you see at least
> two versions of the linux kernel in red hat distributions,
> one with SMP enabled and one without it.
>
> Why not have a python_smp interpreter that allows parallelism
> (and scalability) on SMP machines and another for uniprocessor
> machines where paralellism is not possible?
>
> Yeah, yeah, I know what you're going to say ...
>
> "Please submit a patch."
>
> Sigh, ... if only I had the time ... :-(

A highly educated guess:  it will never happen unless the Python core is
rewritten from scratch.  Reliance on the GIL is ubiquitous now, often in
non-obvious ways.  The last time someone endured the pain of trying was way
back around Python 1.4, when Greg Stein produced a "free threading"
patch/rewrite after much effort.  The fine-grained locking that introduced
slowed Python by about a factor of 2 when running a single thread.  It would
be much more difficult to do this now, because there's much more code now
and much more subtle reliance on the GIL.  There are 100x more third-party
extension modules too now, and apart from the handful specifically written
to increase parallelism, almost all the rest surely rely on the GIL too in
ways the authors remain blissfully unaware of.  Without the GIL, *every*
extension author has to become an expert in threaded programming techniques.
With the GIL, most extension authors can (and do) ignore the possibility of
threads without harm.