HELP! Obscure Python Reentrancy Question

[Tim Peters]

[Courageous]
> I didn't think that Python was structured to allow
> multiple interpreters in a single process?

[Michal Wallace]
> I know python can do this because mod_python uses this
> feature.

Yes, and you can read all about it in the Python/C API Reference Manual,
under Py_NewInterpreter().  Note that it's only available at the C level; no
analogue of this function is exposed at the Python level.

Michal, from the rest of your msg, it sounds like you'd profit more from
reading the "Thread State and the Global Interpreter Lock" section of that
manual.  For straightforward cases it's straightforward <wink>.  For nasty
callbacks it can be a nightmare.

A miserable example in the core is _PyPclose() in posixmodule.c, where a
thread known not to be holding the global interpreter lock needs to obtain
the GIL to call back into the Python runtime.  Normally this is easy:  you
just acquire the GIL.  But to acquire the GIL you first need a valid thread
state, and in this case the thread doesn't have one.  But to create a valid
thread state, you first need a valid interpreter state (this is where
multiple interpreters running in a single process intersects), and in this
case the thread doesn't have one of those either.  Etc.

It took hours to figure out how to make this work, and I had the slight
advantage of knowing what I was doing <wink>.  It's worse if you don't know
whether or not the thread holds the GIL, and worse still if you don't even
know whether Python has been initialized yet.  Mark Hammond's Windows
extensions have to deal with all those cases (_PyPclose is as bad as it gets
in the core), so that's where to look if the only alternative is suicide.