Python Interpreter Reentrancy?

[Dave Kuhlman]

Aahz Maruch <aahz at panix.com> wrote:
> 
> In article <p9r5etcf78818e4ofhu92b1hd5al8ohh5s at 4ax.com>,
> Courageous  <jkraska1 at san.rr.com> wrote:
>>
>>What progress has been made on the reentrancy of the Python interpreter
>>vice the "global interpreter lock"? Is it possible to run python code
>>embedded in a C application with multiple concurrent OS threads, each
>>running time-sliced parallel python scripts, or is that capability
>>still in progress?
> 
> I'm not quite sure what you're asking for.  It's certainly the case that
> multiple threads in a C application can call Python code, but each time
> you call into Python, you need to acquire the GIL first (along with first
> making sure that you've done all the proper initialization).  In theory,
> you could give each thread its own Python interpreter, but that has not
> been done in practice AFAIK (and would suck up a lot of memory).

This has in fact been done in practice.  If I understand what you
mean, we do it in our product.  We use Python as a server-side
scripting language in our Web server add-on.  Each session (one for
each HTTP request) runs in a separate thread.  Each session has its
own Python (sub-) interpreter.  The same sub-interpreter is used to
handle requests from the same end user.  Thus Python maintains name
space and context for us across responses to a user and the
separate (sub-)interpreters keep the context for different sessions
isolated from each other.

This does not, of course, change anything about the global
interpreter lock, which the first question inquired about.  It's
still there.

PyWX for AOLServer makes for an interesting comparison and is
fancier still.  PyWX gives you a (configuration) choice between (1)
multiple Python interpreters, (2) reusing interpreters (from a pool
of sub-interpreters), and (3) running scripts in separate threads
within a single Python interpreter.  I apologize if I'm confusing
things a bit here. I'm still attempting to learn this myself.

These choices have consequences for preserving modules that have
been imported and for preserving the Python global name space.  I'm
still trying to learn those, too.

My understanding is that we (in our product) employ something like
the first option (multiple interpreters), but that we keep that
interpreter alive across responses to the same end user, which is a
little like the second option (reusing interpreters), except that
PyWX clears the name space of each interpreter before reusing it. 
We do not clear the name space, in fact our script writers assume
that the context is preserved.

I believe that an important point here is how much flexibility
Python gives a developer in the way in which Python can be embedded
in another product and how Python is extended with C code and how
scripts are given access to functions in the underlying embedding
application.

  - Dave

-- 
Dave Kuhlman
dkuhlman at rexx.com