[Python-ideas] the future of the GIL

[tomer filiba]

[ moved to python-ideas from python-3000 ]


On 5/8/07, Thomas Heller <theller at ctypes.org> wrote:
> Wouldn't multiple interpreters (assuming the problems with them would be fixed)
> in the same process give the same benefit?  A separate GIL for each one?

hmm, i find this idea quite interesting really:

* builtin immutable objects such as None, small ints, non-heap types,
and builtin functions, would become uncollectible by the GC. after all,
we can't reclaim their memory anyway, so keeping the accounting
info is just a waste of time. the PyObject_HEAD struct would grow
an "ob_collectible" field, which would tell the GC to ignore these
objects altogether. for efficiency reasons, Py_INCREF/DECREF
would still change ob_refcount, only the GC will ignore it for
uncollectible objects.

* each thread would have a separate interpreter, and all APIs should
grow an additional parameter that specifies the interpreter state
to use.

* for compatibility reasons, we can also have a dict-like object mapping
between thread-ids to interpreter states. when you invoke an API,
it would get the interpreter state from the currently executing thread id.
maybe that could be defined as a macro over the real API function.

* the builtin immutable objects would be shared between all instances
of the interpreter. other then those, all other objects would be local
to the interpreter that created them

* extension modules would have to be changed to support
per-interpreter initialization.

* in order to communicate between interpreters, we would use some
kind of IPC mechanism, to serialize access to objects. of course it
would be much more efficient, as no context switches are required
in the same process. this would make each thread basically as
protected as a OS process, so no locks would be required.

* in order to support the IPC, a new builtin type, Proxy, would be added
to the language. it would be the only object that can hold a cross-reference
to objects in different interpreters -- much like today's RPC libs -- only
that wouldn't have to work over a socket.

* if python would ever have a tracing GC, that would greatly simplify
things. also, moving to an atomic incref/decref library could also
help.

of course i'm not talking about adding that to py3k. it's too immature
even for a pre-pep. but continuing to develop that idea more could
be the means to removing the GIL, and finally having really parallel
python scripts.


-tomer