[Python-Dev] multiple interpreters and extension modules
jeremy.kloth at 4suite.org
Fri Dec 22 20:42:11 CET 2006
[[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython
development enough to merit posting here ]]
I have received a bug report for 4Suite that involves a PyObject_IsInstance
check failing for what appears to be the correct class, that is, the class
names match. With some investigating, I have found that the true problem is
with multiple interpreters. The reason for this is that each sub-interpreter
has a "new" copy of any pure Python module. The following issues are also
true for modules that have been reloaded, but I think that is common
knowledge. I mention it only for completeness.
Here is where it crosses into CPython development.
Now extension modules do not get reloaded into each sub-interpreter, but a
copy of the module's dict from when it was first loaded. This particular
extension, when initialized, loads the class in question from the pure Python
module and stores in in a global variable for quick access. I know I can
change this specific case to reload the class each time and for this exact
case, it is most likely what I'll do. However, we have other extensions that
make use of this same technique and their performance would suffer greatly by
doing the lookup for each use of the class.
Also performance isn't the only issue here as we also have C types that
inherit from a Python class and those types will obviously fail an
isinstance() check in Python code run in different interpreters as the base
class is "frozen" at the first import.
So I guess the questions are:
1) is subclassing Python classes in C as a static type supported? Even if they
would be declared on the heap, they would be bound to the first loaded Python
2) would it be worthwhile to have an extension loading function similar to
DllMain() for Windows' DLLs which is called multiple times with a flag
stating the reason for the call? For reference, see:
I envision the reasons as: 'module init', 'module fini', 'interpreter
init', 'interpreter fini'. I see no current need for special treatment of
threads as they share the same interpreter space.
As to question 2, I'm in the process of prototyping that approach to see if it
is feasible in our code and if so, (and people find it useful) I'll attempt
to write a PEP for that feature.
My current thoughts on that are instead of the current init<module name> entry
point in C modules, a new entry point is defined, and if found, used in its
place. That way, this proposal would be optional for extension writers and
existing modules would continue to work without change.
More information about the Python-Dev