[Python-Dev] multiple interpreters and extension modules

[Bob Ippolito]

On 12/23/06, Jeremy Kloth <jeremy.kloth at 4suite.org> wrote:
> On Friday 22 December 2006 5:02 pm, Josiah Carlson wrote:
> > Jeremy Kloth <jeremy.kloth at 4suite.org> wrote:
> > > [[ This may be somewhat c.l.p.-ish but I feel that this crossed into
> > > CPython development enough to merit posting here ]]
> > >
> > > I have received a bug report for 4Suite that involves a
> > > PyObject_IsInstance check failing for what appears to be the correct
> > > class, that is, the class names match.  With some investigating, I have
> > > found that the true problem is with multiple interpreters.  The reason
> > > for this is that each sub-interpreter has a "new" copy of any pure Python
> > > module. The following issues are also true for modules that have been
> > > reloaded, but I think that is common knowledge.  I mention it only for
> > > completeness.
> >
> > If I remember correctly, Python allows you to use multiple interpreters
> > in the same process, but it makes no guarantees as to their correctness
> > when running.
> >
> > See this post for further discussion on the issue:
> > http://mail.python.org/pipermail/python-list/2004-January/244343.html
> >
> > You can also search for 'multiple python interpreters single process' in
> > google without quotes to hear people lament over the (generally broken)
> > multiple Python interpreter support.
>
> The problem here is that it is mod_python using the multiple interpreters.  We
> have no control over that.  What I'm proposing is fixing the extension module
> support for multiple interpreters with the bonus of adding extension module
> finalization which I've seen brought up here before.
>
> Fixing this does require support by the extension module author, but if that
> author doesn't feel the need to work in mod_python (if, of course, they load
> module level constants), that is their loss.
>
> Is 4Suite that different in its use of hybrid Python and C extensions?  There
> is lots of back and forth between the two layers and performance is critical.
> I really don't feel like recoding thousands of lines of Python code into C
> just to get 4Suite to work in mod_python without error.

It's a whole lot more practical to just stop using mod_python and go
for one of the other ways of exposing Python code to the internet. I
bet you can get the same or better performance out of another solution
anyway, and you'd save deployment headaches.

-bob