[Baypiggies] Guido's Blog: It isn't Easy to Remove the

Warren DeLano warren at delsci.com
Sat Sep 15 02:04:11 CEST 2007


Call me crazy, but it seems to me that the problem isn't so much the
GIL, but rather, it is our inability to simultaneously run multiple
Python interpreters within a single process, with each interpreter
playing nicely with native C extensions and libraries.

In other words, it is not the interpreter lock that is so limiting,
rather, it is the fact that the interpreter acts like a global singleton
with respect to the C/API and other native code extensions.

To restate this yet another way:  In my perfect Python-embedded world,
the Python interpreter would itself become an object you can instantiate
many times, and the bulk of the C/API would then become methods called
on that object.  Each interpreter instance would still have its own GIL
-- you wouldn't need to deal with microscopic locking, and you wouldn't
lose any of Python's existing performance.

Of course, most C/API methods would however need to take "self" as an

PyObject *PySome_Action(PythonInterpreterInstance *pythonSelf, ...);

However, we could use TLS (thread-local-state) to dynamically bind a
specific interpreter instance to a specific thread.  

PythonInterpreterInstance *P = ...some extant interpeter instance...;

if(PyInstanceViaTLS_BindThread(P)) {

  PyGILState_STATE gstate;
  gstate = PyGILState_Ensure();

  /* global calls within this thread are now directed at a specific
interpreter */

  PySome_Action (...);
  /* meanwhile, other threads could simultaneously be messaging other
interpreters running in parallel within the same process */



That simple workaround could preserve compatibility will existing C/API
code, while still freeing us from this crushing global singularity...

So is this just crazy talk?  Or might something like this actually be
worth a try? 

warren at delsci.com

More information about the Baypiggies mailing list