Multiple interpreters retaining huge amounts of memory

[Graham Dumpleton]

Nice to see that your comments do come from some understanding of the
issues. Been number of times in the past when people have gone off
saying things about multiple interpreters, didn't really know what
they were talking about and were just echoing what some one else had
said. Some of the things being said were often just wrong though. It
just gets annoying. :-(

Anyway, a few comments below with pointers to some documentation on
various issues, plus details of other issues I know of.

On Feb 3, 6:38 pm, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
> > If you are going to make a comment such as 'multi-interpreter feature
> > doesn't really work' you really should substantiate it by pointing to
> > where it is documented what the problems are or enumerate yourself
> > exactly what the issues are. There is already enough FUD being spread
> > around about the ability to run multiple sub interpreters in an
> > embedded Python application, so adding more doesn't help.
>
> I don't think the limitations have been documented in a systematic
> manner. Some of the problems I know of are:
> - objects can easily get shared across interpreters, and often are.
>    This is particularly true for static variables that extensions keep,
>    and for static type objects.

Yep, but basically a problem with how people write C extension
modules. Ie., they don't write them with the fact that multiple
interpreters can be used in mind.

Until code was fixed recently in trunk, one high profile module which
had this sort of problem was psycop2. Not sure if there has been an
official release yet which includes the fix. From memory the problem
they had was that a static variable was caching a reference to the
type object for Decimal from the interpreter which first loaded and
initialised the module. That type object was then used to create
instances of Decimal type which were passed to other interpreters.
These Decimal instances would then fail isinstance() checks within
those other interpreters.

Some details about this in section 'Multiple Python Sub Interpreters'
of:

  http://code.google.com/p/modwsgi/wiki/ApplicationIssues

That section of documentation also highlights some of the other errors
that can arise where file objects in particular are somehow shared
between interpreters, plus issues when unmarshalling data.

You might also read section 'Application Environment Variables' of
that document. This talks about the problem of leakage of environment
variables between sub interpreters. There probably isn't much that one
can do about it as one needs to push changes to os.environ into C
environment variables so various system library calls will get them,
but still quite annoying that the variables set in one interpreter
then show up in interpreters created after that point. It means that
environment variable separation for changes made unique to a sub
interpreter is impossible.

> - Py_EndInterpreter doesn't guarantee that all objects are released,
>    and may leak. This is the problem that the OP seems to have.
>    All it does is to clear modules, sys, builtins, and a few other
>    things; it is then up to reference counting and the cycle GC
>    whether this releases all memory or not.

There is another problem with deleting interpreters and then creating
new ones. This is where a C extension module doesn't declare reference
counts to static Python objects it creates. When the interpreter is
destroyed and objects that can be destroyed are destroyed, then it may
destroy these objects which are referenced by the static variables.
When a subsequent interpreter is created which tries to use the same C
extension module, that static variable now contains a dangling invalid
pointer to unused or reused memory.

PEP 3121 could help with this by making it more obvious of what
requirements exist on C extension modules to cope with such issues.

I don't know whether it is a fundamental problem with the tool or how
people use it, but Pyrex generated code seems to also do this. This
was showing up in PyProtocols in particular when attempts were made to
recycle interpreters within the lifetime of a process. Other packages
having the problem were pyscopg2 again, lxml and possibly subversion
bindings. Some details on this can be found in section 'Reloading
Python Interpreters' of that document.

> - the mechanism of PEP 311 doesn't work for multiple interpreters.

Yep, and since SWIG defaults to using it, it means that SWIG generated
code can't be used in anything but the main interpreter. Subversion
bindings seem to possibly have a lot of issues related to this as
well. Some details on this can be found in section 'Python Simplified
GIL State API' of that document.

> > Oh, it would also be nice to know exactly what embedded systems you
> > have developed which make use of multiple sub interpreters so we can
> > gauge with what standing you have to make such a comment.
>
> I have never used that feature myself. However, I wrote PEP 3121
> to overcome some of its limitations.

As well as the above there are a number of other issues as well. Ones
I can remember right now are as follows.

First is that one can't use different versions of a C extension module
in different sub interpreters. This is because the first one loaded
effectively gets priority. Am not even sure you get an error when
another interpreter tries to load a different version, it just assumes
the one already loaded is okay. This can mean one may get a set of
Python wrappers which doesn't match the C extension module. In other
words, C extension modules are global to process and not local to sub
interpreter.

I know I have talked about this one numerous times, but can't seem to
see where I cover it in the documentation I pointed at. I'll have to
make sure I add it if it isn't there.

Second issue is that when you call Py_EndInterpreter, it doesn't do
some of the stuff that would be done if it was the main interpreter.
The two main culprits are that it doesn't try to stop non daemonised
Python threads and it doesn't call functions registered with the
atexit module.

One might argue that it shouldn't be calling atexit registered
functions as the process isn't being shutdown, but in Python such
functions being called are really at the point the main interpreter is
being destroyed, not the process. As such, it may be appropriate such
registered functions be called for a specific sub interpreter as well,
but obviously only for callbacks registered in that sub interpreter.

One of the reasons for calling atexit registered functions for a sub
interpreter is to terminate daemonised threads. If one isn't able to
kill off daemonised threads created within a sub interpreter then they
can keep running while and after the sub interpreter has been
destroyed. This could result in just a Python exception occuring for
that thread causing it to exit, but can also cause it to crash the
process.

To ensure proper cleanup of sub interpreters when being destroyed and
allow hosted applications to do things properly on exit they may want
to do, found it necessary to do these two things explicitly, when
possibly the Python internals should provide a means, even if
optional, to do it.

Anyway, have a read through that document as you might find a few
interesting things in there about the current problems. Some stuff
isn't necessarily documented as the code for the package this relates
to just works around the issues so everything works as one would
expected rather. For example the atexit register functions being
called for sub interpreters.

In general what I have found is that as long as you are aware of the
limitations, multiple interpreters are still usable. The one thing I
would avoid is trying to recycle sub interpreters. Once they are
created, only safe thing to do is to destroy them on process exit and
no sooner. Otherwise you get issues that OP is seeing, but also some
of the issues I describe above.

Hope you have find this and the referenced document interesting. :-)

Graham