[Baypiggies] Guido's Blog: It isn't Easy to Remove the

[Warren DeLano]

Hmm...

Call me crazy, but it seems to me that the problem isn't so much the
GIL, but rather, it is our inability to simultaneously run multiple
Python interpreters within a single process, with each interpreter
playing nicely with native C extensions and libraries.

In other words, it is not the interpreter lock that is so limiting,
rather, it is the fact that the interpreter acts like a global singleton
with respect to the C/API and other native code extensions.

To restate this yet another way:  In my perfect Python-embedded world,
the Python interpreter would itself become an object you can instantiate
many times, and the bulk of the C/API would then become methods called
on that object.  Each interpreter instance would still have its own GIL
-- you wouldn't need to deal with microscopic locking, and you wouldn't
lose any of Python's existing performance.

Of course, most C/API methods would however need to take "self" as an
argument:

PyObject *PySome_Action(PythonInterpreterInstance *pythonSelf, ...);

However, we could use TLS (thread-local-state) to dynamically bind a
specific interpreter instance to a specific thread.  

PythonInterpreterInstance *P = ...some extant interpeter instance...;

if(PyInstanceViaTLS_BindThread(P)) {

  PyGILState_STATE gstate;
  gstate = PyGILState_Ensure();

  /* global calls within this thread are now directed at a specific
interpreter */

  PySome_Action (...);
  
  /* meanwhile, other threads could simultaneously be messaging other
interpreters running in parallel within the same process */

  PyGILState_Release(gstate);

  PyInstanceViaTLS_UnbindThread();
}

That simple workaround could preserve compatibility will existing C/API
code, while still freeing us from this crushing global singularity...

So is this just crazy talk?  Or might something like this actually be
worth a try? 

Cheers,
Warren
warren at delsci.com