8 Jul
1999
8 Jul
'99
10:53 p.m.
"TDDM" == The Dragon De Monsyne <dragondm@integral.org> writes:
TDDM> Hmm... A few thoughts.. I've never seen the load
TDDM> problem on my mailman site, even though I run several
TDDM> reasonabally well trafficked lists, HOWEVER, I run a
TDDM> slightly customized version of HyperArch.py which stil.l
TDDM> uses bsddb for data storage.
TDDM> Also, my site does not immedeately archive messages, it runs
TDDM> an archiving cronjob every few hours.(it still dosen't draw
TDDM> much cpu whence the cronjob kicks in, tho)Both those differences (using bsddb and not archiving immediately) explains why you wouldn't see the hit. The bug (now fixed) was in the DumbBTree implementation.
TDDM> really, this archiving system was never meant to be
TDDM> used in the current method of operation (being invoked on
TDDM> each incoming message), and it's design is probably rather
TDDM> non-optimal for this use. It spends substantial time
TDDM> building & tearing down & (un)pickling various structures
TDDM> incurring a bit of overhead.Wonderful understatement Dragon! :)
Running with the patches in the CVS tree, I think the current system can work for a site as heavily trafficked as python.org, but it is very inefficient. Maybe we should document that for high traffic sites, you might want to use an external archiver which only runs from a cronjob. Unfortunately, I haven't done this, so if anybody can contribute a HOWTO, I'd appreciate it.
-Barry