Idea for removing the GIL...

[sturlamolden]

On 8 Feb, 10:39, Vishal <vsapr... at gmail.com> wrote:

> Is it possible that the Python process, creates copies of the
> interpreter for each thread that is launched, and some how the thread
> is bound to its own interpreter ?


In .NET lingo this is called an 'AppDomain'. This is also how tcl
works -- one interpreter per thread. I once had a mock-up of that
using ctypes and Python'c C API. However, the problem with 'app
domains' is that OS handles are global to the process. To make OS
handles private, the easiest solution is to use multiple processes,
which incidentally is what the 'multiprocessing' modules does (or just
os.fork if you are on Unix).

Most people would not consider 'app domains' to be a true GIL-free
Python, but rather think of free threading comparable to .NET, Java
and C++. However, removing the GIL will do no good as long as CPython
uses reference counting. Any access to reference counts must be atomic
(e.g. requiring a mutex or spinlock). Here we can imagine using fine-
grained locking instead of a global interpreter lock. There is a
second problem, which might not be as obvious: In parallel computing
there is something called 'false sharing', which in this case will be
incurred on the reference counts. That is, any updating will dirty the
cache lines everywhere; all processors must stop whatever they are
doing to synchronize cache with RAM. This 'false sharing' will put the
scalability down the drain.

To make a GIL free Python, we must start by removing reference
counting in favour of a generational garbage collector. That also
comes with a cost. The interpreter will sometimes pause to collect
garbage. The memory use will be larger as well, as garbage remain
uncollected for a while and is not immediately reclaimed. Many rely on
CPython because the interpreter does not pause and a Python process
has a small fingerprint. If we change this, we have 'yet another
Java'. There are already IronPython and Jython for those who want
this.


Sturla