8 Jul 1999 8 Jul '99
"TDDM" == The Dragon De Monsyne firstname.lastname@example.org writes:
TDDM> Hmm... A few thoughts.. I've never seen the load TDDM> problem on my mailman site, even though I run several TDDM> reasonabally well trafficked lists, HOWEVER, I run a TDDM> slightly customized version of HyperArch.py which stil.l TDDM> uses bsddb for data storage. TDDM> Also, my site does not immedeately archive messages, it runs TDDM> an archiving cronjob every few hours.(it still dosen't draw TDDM> much cpu whence the cronjob kicks in, tho)
Both those differences (using bsddb and not archiving immediately) explains why you wouldn't see the hit. The bug (now fixed) was in the DumbBTree implementation.
TDDM> really, this archiving system was never meant to be TDDM> used in the current method of operation (being invoked on TDDM> each incoming message), and it's design is probably rather TDDM> non-optimal for this use. It spends substantial time TDDM> building & tearing down & (un)pickling various structures TDDM> incurring a bit of overhead.
Wonderful understatement Dragon! :)
Running with the patches in the CVS tree, I think the current system can work for a site as heavily trafficked as python.org, but it is very inefficient. Maybe we should document that for high traffic sites, you might want to use an external archiver which only runs from a cronjob. Unfortunately, I haven't done this, so if anybody can contribute a HOWTO, I'd appreciate it.