[Mailman-Users] Spurious ImportError stopping the qrunner

[Richard Barrett]

I have a hitherto stable Mailman server currently running MM 2.0.13 with 
Python 2.2.1, both built from source, under Suse 7.3 (kernel 2.4.10-4GB)

When I say stable I mean the server has been running without problems for 
over twelve months, handling light/medium levels of traffic for 450 lists. 
The only interruptions have been for the odd MM or Python or OS update. 
Also an occasional rehearsal of up primary-to-backup switchover procedure.. 
The current configuration has been untouched since August and the only 
recent interruption of service was due to an untidy reboot about 10 days 
ago after we had a total loss of mains power and the UPS batteries ran out 
before the mains came back.

At around 01:30 this morning, the qrunner began failing. It was reported as 
a python ImportError exception when $prefix/Mailman/Utils called form 
$prefix/cron/qrunner executed the code line 'from UserDict import UserDict'

This was a showstopper and not the sort of problem that passes unnoticed 
for long.

I spent several frantic hours of trying figure out what had gone wrong. I 
switched service to our identical backup server (the MM python code and 
data are held on NFS mounts from our Auspex server and we have a rehearsed 
and fairly quick procedure for swapping the primary and backup machine 
identities using DHCP) . No avail. A final last ditch reinstall of both 
Python and Mailman from source code on the current primary server and the 
qrunner was still failing in the same way.

I then stumbled across a UserDict.pyc file in the $prefix/Mailman 
directory; just a .pyc, no matching .py

But, I thought, UserDict.pyc resides in /usr/local/lib/python2.2/, it being 
a standard library module. That UserDict.pyc was in its proper place.

After removing the 'extra' UserDict.py from $prefix/Mailman the qrunner 
started working normally.

During the investigation of the problem I found that, when running python 
from the command line, executing 'from UserDict import UserDict' as 
expected worked if and only if it had not been preceded by an import of 
MM's paths.py which inserts $prefix at the start of the sys.path. Which I 
thought was pretty weird but maybe the 'extra' UserDict.pyc has that effect.

What I cannot explain is why a UserDict.pyc file popped up in the 
$prefix/Mailman directory, presumably in the middle of last night, when the 
service interruption started. Unfortunately, I deleted rather than renamed 
that 'extra' UserDict.pyc so the evidence is gone; my justification is that 
I was in something of a rush to get Mailman back on air before our user 
community lynched me.

I'm inclined to rule out some malicious modification of the system. The MM 
server is dedicated to running Mailman, resides in a locked computer room, 
has a private IP number, resides inside a well structured firewall, only 
gets its mail from our primary mail server after virus checking and I can 
find no sign of tampering with the system from inside or outside our domain.

Has anybody else experienced anything similar to this or have any idea how 
this problem arose?

Any advice or help welcome. I just want to avoid another morning of blind 
panic resulting from this sort of problem.