[Python-Dev] multiple interpreters and extension modules

[Jeremy Kloth]

[[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython 
development enough to merit posting here ]]

I have received a bug report for 4Suite that involves a PyObject_IsInstance 
check failing for what appears to be the correct class, that is, the class 
names match.  With some investigating, I have found that the true problem is 
with multiple interpreters.  The reason for this is that each sub-interpreter 
has a "new" copy of any pure Python module. The following issues are also 
true for modules that have been reloaded, but I think that is common 
knowledge.  I mention it only for completeness.

Here is where it crosses into CPython development.

Now extension modules do not get reloaded into each sub-interpreter, but a 
copy of the module's dict from when it was first loaded.  This particular 
extension, when initialized, loads the class in question from the pure Python 
module and stores in in a global variable for quick access. I know I can 
change this specific case to reload the class each time and for this exact 
case, it is most likely what I'll do.  However, we have other extensions that 
make use of this same technique and their performance would suffer greatly by 
doing the lookup for each use of the class.

Also performance isn't the only issue here as we also have C types that 
inherit from a Python class and those types will obviously fail an 
isinstance() check in Python code run in different interpreters as the base 
class is "frozen" at the first import.

So I guess the questions are:
1) is subclassing Python classes in C as a static type supported? Even if they 
would be declared on the heap, they would be bound to the first loaded Python 
class.

2) would it be worthwhile to have an extension loading function similar to 
DllMain() for Windows' DLLs which is called multiple times with a flag 
stating the reason for the call? For reference, see:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp
I envision the reasons as: 'module init', 'module fini', 'interpreter 
init', 'interpreter fini'. I see no current need for special treatment of 
threads as they share the same interpreter space.

As to question 2, I'm in the process of prototyping that approach to see if it 
is feasible in our code and if so, (and people find it useful) I'll attempt 
to write a PEP for that feature.

My current thoughts on that are instead of the current init<module name> entry 
point in C modules, a new entry point is defined, and if found, used in its 
place. That way, this proposal would be optional for extension writers and 
existing modules would continue to work without change.

Feedback appreciated.

-- 
Jeremy Kloth
http://4suite.org/