Just a quick note 'cause I have very little time. I'm currently seeing python.org massively pegged, and Guido and I were talking about some Python tools we'd like to develop that would help debug situations like this. What I wanted was something like gdb's ability to attach to and print stack traces of running external programs. We got into some brainstorming and came up with A Certified Very Cool Trick[1].
This yielded a traceback for where at least two pegged processes are spinning. Seems to make sense, but I'm not very familar with the archiving guts, so I post this traceback to spur some discussion. Maybe Scott or Harald can craft a fix.
Hmm... A few thoughts.. I've never seen the load problem on my mailman site, even though I run several reasonabally well trafficked lists, HOWEVER, I run a slightly customized version of HyperArch.py which stil.l uses bsddb for data storage.
Also, my site does not immedeately archive messages, it runs an archiving cronjob every few hours.(it still dosen't draw much cpu whence the cronjob kicks in, tho)
really, this archiving system was never meant to be used in thecurrent method of operation (being invoked on each incoming message), and it's design is probably rather non-optimal for this use. It spends substantial time building & tearing down & (un)pickling various structures incurring a bit of overhead.
-The Dragon De Monsyne