[Python-Dev] Why is the GIL not in PyInterpreterState?

[Tobias Oberstein]

Thanks for replying, I now see the kinds of problems much clearer:

1. global interpreter/VM state
2. global state in object implementations
3. global state in extension modules
4. shared process resources (file descriptors,..)

.. that is

1,2  could in principle be fixed, but heavy work
3    unlikely ever (for all modules)
4    not a bug, but a feature


> OTOH, if your goal is N completely independent interpreters, why 
> not fire up
> N processes?  Then independence comes for free and you don't have 
> to change
> a thing.  If the answer is that you don't really want them to be
> *completely* independent, then maintaining the refcounts on the 
> objects you
> want them to share will be a nightmare:  every Py_INCREF and Py_DECREF in
> the codebase also relies on the process-level GIL for proper operation.
> 

I only want to share state via objects of a special extension class X,
which wraps up an OODBMS (which is the embedding application) and takes
care of the necessary synchronisation on it's own.

X should be subclassable in Python.

I want to do it within a single process, because I don't want a
OS context switch on every method call to objects of class X.

In a perfect world, problems 1,2 of above would be fixed, I would
only use "nice" modules which obey 3 of above, won't care about 4 and
would end up with either

a) each VM has exactly one OS thread

or

b) each VM has at least on OS thread

Thus, a) required no locking at all and b) required a "global" lock per VM.

Greets,
Tobias