Mailman 3 July 2004 - Mailman-Developers

VERPing: ouch!
by Fil 14 Jul '04

14 Jul '04

Hi, I think there's a small design flaw with the once-in-a-wile VERPing scheme. My biggest list is 180k subscribers, and I've set up Mailman to VERP once every 10th message. Well, it happened today that the big list was hit by its VERP time, and it's a bit awful - it looks like the list has taken control of Mailman, and no other mail can pass through it. I think it might be wiser to say "every 10th recipient" is VERPed at each sending, than what we have now. Of course, it needs rotating things, or randomizing them, so it's not a two-lines of code I guess. Meanwhile I'll just have to disable VERP for this list (or for the whole server). -- Fil

7 17

Re: [Mailman-Developers] How to export memberlist to file
by J C Lawrence 14 Jul '04

14 Jul '04

On Tue, 13 Jul 2004 17:24:11 +0200 Nick vd Kloor <@ FOR-Nation" <nick.vdkloor(a)for-nation.com>> wrote: > Hi all, not sure anymore if there is a function or parameter to get > your whole memberlist in a textfile. Does anyone know? ~/bin/list_members. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 14 Jul '04

14 Jul '04

On Tue, 13 Jul 2004 17:33:43 +0200 fil <fil(a)rezo.net> wrote: >> I thought v2 chunked the queue processing... Hurm, if it doesn't it >> should and that's an easy enough fix. >> From what I read we're in SMTPDirect.py : > def process(mlist, msg, msgdata): > this function does process all recipients (in chunks if not VERP) > consecutively, before exiting. We spent 5 hours in its main loop for > that one message. The painful bit is in Runner.py which appears to linearly process each message queue. The next optimisation would have queue runners forking off every N minutes, with each one opportunistically handling messages in the queue. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 13 Jul '04

13 Jul '04

On Tue, 13 Jul 2004 17:22:26 +0200 fil <fil(a)rezo.net> wrote: > Yes, the MTA *was* twiddling its thumbs, and the system too... and as > Mailman's daemon was sending one VERP msg at a time, it was not > processing the usual incoming stuff, and not delivering the other > lists. v2.x or v1.x? I thought v2 chunked the queue processing... Hurm, if it doesn't it should and that's an easy enough fix. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 13 Jul '04

13 Jul '04

On Tue, 13 Jul 2004 14:28:34 +0200 fil <fil(a)rezo.net> wrote: >> I ask because Chuq Von Rospach has done some calculations on what >> should theoretically happen to your performance if you enable VERP, >> but I don't know of anyone who has actually timed the performance >> difference on large lists. > Usually the sending (mailman to postfix to 90% of users) takes a bit > more than two hours ; yesterday it took about 6 hours. That's an effective delivery rate of ~1,300 deliveries per minute, which is fairly low assuming moderate hardware. ObNote: I found I could sustain 2,400 deliveries per minute with a quick-poke tuned Postfix on a 512Meg RAM PII-333 with separate spindles for spool and log. Some minor efforts at more disciplined tuning suggested that it could sustain 2,800+, but I could never verify that as I just couldn't keep the spool fed fast enough. > But more importantly, the Mailman -> postfix thing took 5 hours > instead of ~ 15 minutes. What system metrics spiked during this time? 160K/5 hours is a delivery rate of less than 600 per minute. That's one message every 10+ seconds which is quite slow. Even if you double that (receipt and transmission) that's only 1,200 per minute which is still dawdling territory. Your MTA should have been able to keep that spool empty while twiddling its thumbs. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

3 2

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 13 Jul '04

13 Jul '04

On Tue, 13 Jul 2004 10:44:22 -0400 Bob Puff <bob(a)nleaudio.com> wrote: > Any FAQ on your "quick-poke" tunings? I don't recall anything off the top of my head that's not in the list archives or the User FAQ (I reported on it at the time in both places IIRC). I've not run Postfix for some years now (I'm currently Exim-everywhere due to UID/GUI control and message rewriting issues) and so would have to re-research the docs to give a more accurate answer. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 13 Jul '04

13 Jul '04

On Tue, 13 Jul 2004 15:26:58 +0200 Brad Knowles <brad.knowles(a)skynet.be> wrote: > I will definitely update the VERP performance entry in the FAQ to > reference your experience. I wouldn't, not yet. The numbers are too far off from reasonable expectation. I'd bet there are other strongly unknown factors which are impeding performance (no local cacheing name server, spool and log on same ATA spindle, something). -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 13 Jul '04

13 Jul '04

On Tue, 13 Jul 2004 14:44:11 +0200 fil <fil(a)rezo.net> wrote: >> Here's the smtp log line: Jul 13 02:05:22 2004 (435) >> <20040712150834.DCA153680BD(a)alan.rezo.net> smtp for 152991 recips, >> completed in 31782.241 seconds > .../... >> In any case, 2 seconds per VERP message for such a big list is too >> costly, > Someone kindly told me this makes only 0.2s /message, not 2 seconds $ calc "152991 / 31782.241" ~4.81372600503532774797 -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

Re: [Mailman-Developers] VERPing: ouch!
by J C Lawrence 13 Jul '04

13 Jul '04

On Tue, 13 Jul 2004 13:52:16 +0200 Brad Knowles <brad.knowles(a)skynet.be> wrote: > This is really more of an MTA limitation, although there might be some > things we can do to try to work around it with Mailman. For example, > it might be faster/lower overall load on the server if we had the MTA > do the VERPing for us -- we're pretty sure that's supported by some > MTAs (e.g., at least some versions of Exim), and we know it's faster > for at least some of them (e.g., Exim). s/Exim/QMail/g Exim does have some supports, but not to the degree that QMail does, and not to the performance degree that QMail does. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1 0

AssertionError in 2.1.5
by Tom G. Christensen 12 Jul '04

12 Jul '04

Hi, I just hit a bug in 2.1.5 in the 'Confirm subscription request' page. If you press 'Cancel my subscription request' you just get a crash like the one below. Just as an experiment I tried the same thing with my subscription to mailman-developers and the Mailman 2.1.5 at mail.python.org promptly crashed with the same Traceback. Bug in Mailman version 2.1.5 We're sorry, we hit a bug! If you would like to help us identify the problem, please email a copy of this page to the webmaster for this site with a description of what happened. Thanks! Traceback: Traceback (most recent call last): File "/export/home/mailman/scripts/driver", line 87, in run_main main() File "/export/home/mailman/Mailman/Cgi/confirm.py", line 114, in main subscription_cancel(mlist, doc, cookie) File "/export/home/mailman/Mailman/Cgi/confirm.py", line 312, in subscription_cancel userdesc = mlist.pend_confirm(cookie)[1] File "/export/home/mailman/Mailman/Pending.py", line 141, in pend_confirm assert self.Locked() AssertionError Python information: Variable Value sys.version 2.3.4 (#1, Jun 2 2004, 10:42:24) [GCC 3.3.2] sys.executable /usr/local/bin/python2.3 sys.prefix /usr/local sys.exec_prefix /usr/local sys.path /usr/local sys.platform sunos5 <environment snipped> -tgc

2 2