GIL in the new glossary

[Harri Pesonen]

Luis P Caamano wrote:

> The glossary at <http://www.python.org/dev/doc/devel/tut/node16.html>
> has the following definition for the GIL:
> 
>>global interpreter lock 
>>
>>The lock used by Python threads to assure that only one thread
>>can be run at a time. This simplifies Python by assuring that no
>>two processes can access the same memory at the same time. Locking
>>the entire interpreter makes it easier for the interpreter to be
>>multi-threaded, at the expense of some parallelism on
>>multi-processor machines. 
> 
> Some parallelism???  Wouldn't it be more accurate to say
> "at the expense of parallelism?"  The GIL doesn't eliminate
> "some" paralellism, it completely eliminates any chance of
> parallelism within the same interpreter.

I agree, GIL sucks.

>>Efforts have been made in the past to create a "free-threaded"
>>interpreter (one which locks shared data at a much finer
>>granularity), but performance suffered in the common
>>single-processor case. 
> 
> This is true for kernels too, which is why you see at least
> two versions of the linux kernel in red hat distributions,
> one with SMP enabled and one without it.
> 
> Why not have a python_smp interpreter that allows parallelism
> (and scalability) on SMP machines and another for uniprocessor
> machines where paralellism is not possible?
> 
> Yeah, yeah, I know what you're going to say ... 
> 
> "Please submit a patch."
> 
> Sigh, ... if only I had the time ... :-(

Exactly. I have began work on this. I call it Viper, because we don't 
have Pythons in Finland. It takes the latest Python C API source code 
and makes it GIL free. It is a Python script.

I have just began the work, currently it just removes the deallocator 
from static objects like Py_None so that it does not matter what the 
reference count is. Static objects will be never deallocated, so they 
are thread-safe.

Next step is to add the pointer to thread state to every C API function, 
so that there is no global state anymore. This will remove GIL, or 
actually convert it to LIL, local interpreter lock. LIL is still needed 
if you want to create threads in Python code. This steps makes it 
possible to create several free-threading interpreters in the same process.

The final step is to remove LIL, but this breaks the existing Python 
threading code, so something substituting should be developed. I had an 
idea of shared interpreter state for shared memory access. Or separate 
functions for inter-thread communications.

Harri