[Python-Dev] GIL, Python 3, and MP vs. UP

[Phillip J. Eby]

At 06:10 PM 9/20/2005 -0400, Bob Ippolito wrote:
>My use case for this isn't so much about threads, but plug-ins.
>Writing multiple Python-based plug-ins for an application is always a
>mess, because they share too much (sys.modules, sys.path, etc.).
>PyObjC would benefit greatly from this feature, because you can write
>Python-based plug-ins for any Cocoa app that supports plug-ins, even
>if they're otherwise unaware of Python's existence.  There are
>workarounds, of course, with import hooks and similar hacks.  I think
>that mod_python would also benefit from this, and probably other such
>systems.

To that list of shared things, one can easily add sys.argv, sys.stdin, 
sys.stdout, sys.stderr, os.environ, and on and on and on.  But the ones you 
mention present special issues for testing, especially testing of import 
hooks, path management tools and so on.  Decimal contexts are slightly 
better in that they're at least thread-local.  But even if all of these 
things were thread-local, you'd still have problems with task switches 
between pseudothreads, or just maintaining logically separate contexts 
between e.g. different plugins.

I've actually got an idea of how to solve these problems, at least for 
Python-level code, although for it to work right with Python's import 
internals it'd be necessary to use PyObject_* APIs against sys.modules 
instead of PyDict_* ones.  Apart from that, it's implementable in pure 
Python, so non-CPython implementations should be able to use it too.

I've been considering writing a PEP for it, since it also adds a 
generally-useful way of dealing with "context variables" (like the Decimal 
context and the sys.* variables), while being able to support switching 
*all* context variables simultaneously and instantaneously when changing 
execution contexts (like switching between coroutines).  For example, it 
would let 1000 pseudothreads each have a unique current value for 
sys.stdout or the current decimal context, addressing a gap in the fit 
between PEP 343 and PEP 342.  There's currently no reasonable way to task 
switch between co-routines in the body of a 'with:' statement, so things 
like "with redirecting_stdout()" to two places in different coroutines 
breaks horribly.  But context managers implemented via context variables 
would handle such situations automatically, and with isolation of effects 
to the current thread.  It would also allow you to have logically distinct 
plugins by swapping their context variables in when you make callbacks to 
them, so the notion of the "task" to which a set of context variables 
applies is not limited to coroutines.

My prototype implementation of the basic idea is <200 lines of Python and 
very fast: ~2usec on my budget PC to switch between two arbitrarily-sized 
sets of context variables.  The data structure is O(1) for switching sets 
or changing values; it's only O(N) the next time you change a value after 
taking a read-only snapshot of the whole context.  (Taking a snapshot is 
O(1) due to copy-on-write; you can take as many snapshots of the same 
unchanged state without creating any new objects.)

It also takes about 2uS to read a context variable, alas, but a C version 
could drop the overhead down to as little as a single dictionary lookup if 
the "context pointer" were kept in the thread state structure.  (A fair bit 
of the Python implementation's overhead is just getting to the context 
mapping for the current thread.)

The idea isn't fully PEP-ready at this point, though.  I still need to 
flesh out my ideas regarding how context variables would be initialized in 
new threads, for example, or when you spawn new coroutines.