[Python-bugs-list] [Bug #110601] expert on extension modules and multiple interpreters? (PR#125)

[noreply@sourceforge.net]

Bug #110601, was updated on 2000-Jul-31 14:05
Here is a current snapshot of the bug.

Project: Python
Category: Core
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Summary: expert on extension modules and multiple interpreters? (PR#125)

Details: Jitterbug-Id: 125
Submitted-By: Nikolas Kauer <kauer@pheno.physics.wisc.edu>
Date: Tue, 9 Nov 1999 18:43:09 -0600 (CST)
Version: None
OS: None

Dear Guido,

    (please read first paragraph)

I discovered a Python problem that so far nobody could really explain, 
since a precise explanation requires in-depth knowledge about the internals 
of the Python interpreter.  I HAVE DONE MY HOMEWORK and have posted this 
message to various mailing lists including comp.lang.python, python-help@python.org 
and pyapache-devel@msg.com.mx and communicated with several people individually, 
getting some guesses (appended after the problem description), but no competent answer, 
yet.  I'm sure you know somebody who could provide that answer, and would appreciate 
it if you could FORWARD THIS MESSAGE to such a person.

Best regards,
Nikolas

PS I'm sure you heard this a thousand times, nevertheless I like Python 
and enjoy programming in Python.  I think it's a beautifully designed 
programming language.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I run the following test script on Apache-1.3.6/PyApache-4.19
on Linux 2.2.5 (RedHat 6.0) with only one httpd child and the
module mxDateTime compiled into Python/PyApache, so there is
no dynamic loading of a shared library.

------------------------------------------------------------
print "Content-type: text/plain"
print
print
import sys, traceback
# just checking ...
if not sys.modules.has_key('mxDateTime'):
   print 'No module mxDateTime in sys.modules'
   print

import mxDateTime
print "sys.modules['mxDateTime'] = ", sys.modules['mxDateTime']
print "mxDateTime.__dict__['now'] = ", mxDateTime.__dict__['now']
print
print "calling mxDateTime.now() gives:",
try:
   print mxDateTime.now()
except:
   print
   traceback.print_exc(file=sys.stdout)
--------------------------------------------------------------
First time http://localhost/mytest.py gives
*******************************
No module mxDateTime in sys.modules

sys.modules['mxDateTime'] =  <module 'mxDateTime' (built-in)>
mxDateTime.__dict__['now'] =  <built-in function now>

calling mxDateTime.now() gives: 1999-10-28 18:28:49.75
*******************************
And again http://localhost/mytest.py now gives
*******************************
No module mxDateTime in sys.modules

sys.modules['mxDateTime'] =  <module 'mxDateTime' (built-in)>
mxDateTime.__dict__['now'] =  <built-in function now>

calling mxDateTime.now() gives:
Traceback (innermost last):
  File "/local/web/alpha/docroot/mytest.py", line 16, in ?
    print mxDateTime.now()
TypeError: call of non-function (type None)
*******************************
After that experience, I eliminated all dependencies on module
DateTime/mxDateTime in my Python code, but now I get
the same error whenever something in module MySQLdb is
accessed for the second time ...

The mxDateTime author Marc Lemburg <mal@lemburg.com> writes:
"[...] indicates that PyApache (or perhaps the Python finalizer)
has cleared the module's namespace... which is bad, since
extension modules can typically only initialize themselves *once*."
Marc writes further:
"[T]his is really odd: the 'now' symbol still refers to the
existing function while the module seems to return None as
attribute."

The PyApache coordinator Lele Gaifax <lele@seldati.it> writes:
"I can confirm that the problem lives in the cleanup mechanism.
Python clears up its dictionaries, and doesn't allow a
reinit, as Marc pointed out.  This is not a PyApache fault: you
can get the very same result running the pysrv demo [...].
In fact, every use of such modules mixed to multiple interpreters
should cause the problem."

Apparently, the fact that a module works with regular Python
does not guarantee that it also works with PyApache (or pysrv).

Lele Gaifax writes:
"[...]I cannot see a way out, if not digging in Python's internals.
[D]oes someone know if the problem has been raised before in the
Python community?"

Please, could someone with Python internals expertise explain this
issue?  How can I tell in advance if an extension module will work
with PyApache?  Is it possible to trick Python into cleanly
re-initializing a module?  Is there any hope that I can run my
existing Web site with the Python interpreter embedded into
Apache's httpd?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

More insights from this past week:

Lele Gaifax writes:
"Well, I seems to have tracked it down to the cache you use on Python's 
time module. I changed the function mxDateTime_now, where's the only
reference to that module, to use `PyImport_AddModule ("time")'
instead and extracting the dictionary from its result instead from
Python_time_module, that I left alone.

That function returns a borrowed reference to the module object,
*provided* it is already loaded. Since this is done at mxDateTime's
initialization, this is probably safe, but it needs to be checked...

Anyway, so patched the module does not break PyApache, [...]"

Marc Lemburg replies:
"That's interesting: the time module's namespace is probably cleared
during the Fini code stage. This would change the interpretation
of the error. What you see is the error produced by the internals
of the now() function and not related to the mxDateTime module
namespace itself.

Since time.time is the only function I really use from the
Python time module perhaps caching only the function would do
the trick.

BTW, when finalization occurrs, is mxDateTime_Cleanup() being
called ? It should reset the global reference to the time
module (at least that's what my current code does):

[code]

Something else I could do is move time module lookup from
the initmxDateTime() function into now() itself in case that
helps.
 
[...]

I would guess that the setup mxODBC + mxDateTime has the
same problems related to mxODBC holding a reference to mxDateTime."

Any help in form of diagnostics, explanations and suggestions would be 
greatly appreciated.

Nikolas

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Some ideas from python-help@python.org:

Mark Hammond <mhammond@skippinet.com.au> writes:
"If I understand the problem correct, it is simply that Python can not
handle multiple Py_Init()/Py_Finalize() calls.  Doing a Py_Finalize()
cleans things up, but another Py_Init() can not restore everything
back to a working state.

I agree this is a problem, and it has ben brought up before.  I suffer
the same problem with the COM extensions.  Ideally this needs to be
fixed properly, but it is not trivial to do so - the entire module
init process would need to be re-thought.  The only solution I can
think of is to get Apache to never finalize Python.

Of course, I may be completely off-track here..."

Martin von Loewis <loewis@informatik.hu-berlin.de> writes:
"I think the problem is different. PyApache does *not* invoke
Py_Finalize(*), but Py_EndInterpreter. This does PyImport_Cleanup,
then PyInterpreterState_Delete.

This is where I got stuck: The PyImport_Cleanup iterates over the
interpreter-state's module list, which is then cleared, and completely
discarded. I did not actually find the place where state is preserved
across interpreters, so the problem can't possibly happen :->

(*) Of course, Apache eventually does Py_Finalize, when the whole
server is shut down. Each individual CGI only goes through
Py_NewInterpreter/Py_EndInterpreter."




====================================================================
Audit trail:
Fri Dec 03 10:21:06 1999	guido	changed notes
Fri Dec 03 10:21:07 1999	guido	moved from incoming to open
Fri Dec 03 10:21:47 1999	guido	changed notes

Follow-Ups:

Date: 2000-Aug-13 23:31
By: mhammond

Comment:
Assigning to MAL - it involves mxODBC and Apache on a Linux box, so surely he is more qualified :-)  I'm inclined to mark this is irreproducible, but maybe Marc has had further insights.
-------------------------------------------------------

Date: 2000-Aug-14 03:31
By: lemburg

Comment:
The problem you describe is an artifact of the way mxDateTime
tries to reuse the time.time() API available through the
standard Python time module. I think I have fixed the problem
in my latest version. If you're interested I can email you a
pre-release copy to test it against PyApache.

-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=110601&group_id=5470