[Python-Dev] relative package imports, version numbering, yadda yadda

[Skip Montanaro]

I haven't been paying real close attention to the relative package import
thread on python-dev, but some way or other I strongly believe something has
to be done to improve the situation I now find myself in or more and more
people are going to start getting bitten by the same sort of problem.  Allow
me to 'splain.

I use a web server based on Zope's ZServer (which is itself based on medusa)
sitting behind an Apache web server as a long-running process to handle what
many people traditionally used CGI for.  It's a big performance boost over
CGI.  Unfortunately, I have recently been experiencing frequent server
hangs.  So far I've been unable to figure out what the cause is.  I do
notice occasional tracebacks like:

    Unhandled exception in thread:
    Traceback (innermost last):
      File "ZServer/PubCore/ZServerPublisher.py", line 97, in __init__
      File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish
	self.stdout.close()
      File "/home/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close
	self._channel.push(CallbackProducer(self._channel.done))
      File "/home/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push
	if send: self.initiate_send()
      File "/usr/lib/python1.5/asynchat.py", line 199, in initiate_send
	_push_mode = 0
      File "/usr/lib/python1.5/asynchat.py", line 191, in refill_buffer
	self.ac_in_buffer = ''
      File "/usr/lib/python1.5/asynchat.py", line 253, in pop
	# this could maybe be made faster with a computed regex?
    IndexError: list index out of range

and also notice that the server can pile up a huge number of connections in
the ESTABLISHED state, at which point the whole mess grinds to a halt with
not much computation or network traffic happening.  A separate shell script
uses netstat to detect when a large number of sockets have piled up and
restarts the server.  Brutal, but crudely effective. (When in doubt, treat
the symptoms...)

Today it just dawned on me looking at the above traceback that ZServer is
getting the wrong version of asynchat.py (and presumably of asyncore.py as
well).  It's using the version delivered with Python 1.5.2 distribution
instead of the version that was delivered with the version of Zope I'm using
(1.something).

So I started looking around at the versions and dates of various copies of
asynchat.py.  Here's what I found:

    source		version number:date	owner
    Python 1.5.2	1.2:1999/06/18		guido
    Python CVS		1.2:1999/06/18		guido
    Zope 1.?		1.7:1999/04/09		amos
    Zope 2.0		1.9:1999/07/19		amos
    Medusa 990902	2.24:1999/07/07		rushing

What's apparently been happening is that people have picked up asyncore and
asynchat at various time and stuck them in their own CVS repositories
without somehow freezing the Id string of the version they originally got
from Sam Rushing.  It's not clear what the differences are until you compare
the actual files.  It turns out that the Zope 2.0 and Medusa versions have
no content differences, only wildly different version numbers.  The Medusa
and Python CVS versions only have one difference:

    if index > 0:
    	# don't bother reporting the empty string (source of subtle bugs)
    	self.collect_incoming_data (self.ac_in_buffer[:index])

which *may* be what's causing my problems (note the IndexError in my
traceback).

Somewhere along the way I think we need to apply more restraint to our
packaging and numbering of Python modules.  I don't know what form that
restraint will eventually take, but at this point I'm willing to replace
Greg Ward's s/must/should/ with s/should/must/.

a-nightmare-indeed!-i don't think-this-is-what-Sam-meant-ly y'rs

Skip Montanaro | http://www.mojam.com/
skip@mojam.com | http://www.musi-cal.com/~skip/
847-971-7098   | Python: Programming the way Guido indented...