extension modules and multiple interpreters

[nkauer at my-deja.com]

I run the following test script on Apache-1.3.6/PyApache-4.19
on Linux 2.2.5 (RedHat 6.0) with only one httpd child and the
module mxDateTime compiled into Python/PyApache, so there is
no dynamic loading of a shared library.

------------------------------------------------------------
print "Content-type: text/plain"
print
print
import sys, traceback
# just checking ...
if not sys.modules.has_key('mxDateTime'):
   print 'No module mxDateTime in sys.modules'
   print

import mxDateTime
print "sys.modules['mxDateTime'] = ", sys.modules['mxDateTime']
print "mxDateTime.__dict__['now'] = ", mxDateTime.__dict__['now']
print
print "calling mxDateTime.now() gives:",
try:
   print mxDateTime.now()
except:
   print
   traceback.print_exc(file=sys.stdout)
--------------------------------------------------------------
First time http://localhost/mytest.py gives
*******************************
No module mxDateTime in sys.modules

sys.modules['mxDateTime'] =  <module 'mxDateTime' (built-in)>
mxDateTime.__dict__['now'] =  <built-in function now>

calling mxDateTime.now() gives: 1999-10-28 18:28:49.75
*******************************
And again http://localhost/mytest.py now gives
*******************************
No module mxDateTime in sys.modules

sys.modules['mxDateTime'] =  <module 'mxDateTime' (built-in)>
mxDateTime.__dict__['now'] =  <built-in function now>

calling mxDateTime.now() gives:
Traceback (innermost last):
  File "/local/web/alpha/docroot/mytest.py", line 16, in ?
    print mxDateTime.now()
TypeError: call of non-function (type None)
*******************************
After that experience, I eliminated all dependencies on module
DateTime/mxDateTime in my Python code, but now I get
the same error whenever something in module MySQLdb is
accessed for the second time ...

The mxDateTime author Marc Lemburg <mal at lemburg.com> writes:
"[...] indicates that PyApache (or perhaps the Python finalizer)
has cleared the module's namespace... which is bad, since
extension modules can typically only initialize themselves *once*."
Marc writes further:
"[T]his is really odd: the 'now' symbol still refers to the
existing function while the module seems to return None as
attribute."

The PyApache coordinator Lele Gaifax <lele at seldati.it> writes:
"I can confirm that the problem lives in the cleanup mechanism.
Python clears up its dictionaries, and doesn't allow a
reinit, as Marc pointed out.  This is not a PyApache fault: you
can get the very same result running the pysrv demo [...].
In fact, every use of such modules mixed to multiple interpreters
should cause the problem."

Apparently, the fact that a module works with regular Python
does not guarantee that it also works with PyApache (or pysrv).

Lele Gaifax writes:
"[...]I cannot see a way out, if not digging in Python's internals.
[D]oes someone know if the problem has been raised before in the
Python community?"

Please, could someone with Python internals expertise explain this
issue?  How can I tell in advance if an extension module will work
with PyApache?  Is it possible to trick Python into cleanly
re-initializing a module?  Is there any hope that I can run my
existing Web site with the Python interpreter embedded into
Apache's httpd?

Nikolas


Sent via Deja.com http://www.deja.com/
Before you buy.