[Python-3000] Kill GIL?

[Josiah Carlson]

"Andre Meyer" <meyer at acm.org> wrote:
> Dear Python experts
> 
> As a heavy user of multi-threading in Python and following the current
> discussions about Python on multi-processor systems on the python-list I
> wonder what the plans are for improving MP performance in Py3k. MP systems
> become more and more common as most modern processors have multiple
> processing units that could be used in parallel by distributing threads.
> Unfortunately, the GIL in CPython prevents to use this mechanism. As far as
> I understand IronPython, Jython and PyPy do not suffer from this.
> 
> While I understand the difficulties in removing the GIL and the potential
> negative effect on single-threaded applications I would very much encourage
> discussion to seriously consider removing the GIL (maybe optionally) in
> Py3k. If not, what alternatives would you suggest?

Search for 'Python free threading' without quotes in Google to find the
discussions about this topic over the years.

Personally, I think that the effort to remove the GIL in Py3k (or
otherwise) is quite a bit of trouble that we don't want to have to go
through; both from an internal redesign, and C-extension perspective.

It would be substantially easier if there were a distributed RPC
mechanism that auto distributed to the "least-working" process in a set
of potential working processes on a single machine.  Something with the
simplicity of XML-RPC calling (but without servers and clients) and the
distribution properties of Linda. Of course then we run into a situation
where we need to "pickle" the callable arguments across a connection of
some kind. There is a solution to this on a single machine; copying the
internal representation of every object in the arguments of a function
call to memory shared between all processes (mmap).  With such a
semantic, only mutable portions need to be copied out into non-mmap
memory.

With that RPC mechanism and file handle migration (available on BSDs
natively, linux with minor work, and Windows via pywin32), most
operations would *just work*, would be reasonably fast, and Python could
keep its GIL - which would be substantially less work for everyone
involved.


The details are cumbersome (specifically the copying Python objects
to/from memory), but they can be made less cumbersome if one only allows
builtin objects to be transferred.

 - Josiah