Mailman 3 November 2000 - Mailman-Developers

what happens after writing to qfiles?
by Andreas Gietl Nov. 6, 2000

Nov. 6, 2000

Hi, since a few weeks i am exploring mails not proceeded by mailman after upgrades to rc1 or even on servers with a new fresh installation. Since it seems you could not help me till today i want to ask you what happens after mailman wrote the two files to qfiles when it gets a confirmation message? How are they proceeded? concretly: what happens after this is done? [pid 32132] open("/home/mailman/qfiles/5d8a73f7dfa649d60b947426c439a34c2b17f4b9.db", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 32132] umask(0) = 022 [pid 32132] open("/home/mailman/qfiles/5d8a73f7dfa649d60b947426c439a34c2b17f4b9.db", O_WRONLY|O_CREAT|O_TRUNC, 0664) = 5 [pid 32132] fcntl(5, F_GETFL) = 0x1 (flags O_WRONLY) [pid 32132] fstat(5, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 [pid 32132] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4014a000 [pid 32132] _llseek(5, 0, [0], SEEK_CUR) = 0 [pid 32132] umask(022) = 0 [pid 32132] write(5, "{s\7\0\0\0versioni\2\0\0\0s\t\0\0\0torequest"..., 124) = 124 [pid 32132] close(5) = 0 [pid 32132] munmap(0x4014a000, 4096) = 0 [pid 32132] umask(0) = 022 [pid 32132] open("/home/mailman/qfiles/5d8a73f7dfa649d60b947426c439a34c2b17f4b9.msg", O_WRONLY|O_CREAT|O_TRUNC, 0664) = 5 thx andreas -- andreas gietl gietl internet services fon +49 9402 2551 fax +49 9402 2604 mobile +49 171 60 70 008 gietl(a)gietl.com ############################################ # Das Handbuch sagt, das Programm benötige # # Windows 95 oder besser. Also habe ich # # Linux installiert! # ############################################

1 1

Re: [Mailman-Developers] Re: [Mailman-Users] Problem with qrunner and too much incoming mail
by Nigel Metheringham Nov. 6, 2000

Nov. 6, 2000

[Please can we try and kill these postings that are sent to both mailman-users & mailman-developers] Having been watching the floods at home in York for the last few days I'll just come in at this point chuqui(a)plaidworks.com said: > Nope. This is a job for -- ta da -- an API with a plug in > architecture. Don't try to solve the huge list probelm, but solve the > 90% problem (what works for the first 90% of the sites out there), > and make it easy to plug in a different delivery beast so that other > 10% can simply write a new module and swap it in (and hopefully send > it back to the mailman project...) [Loud applause] Mailman is a MLM, its not an MTA. Don't think about making it an MTA. Just leave enough hooks for people to trash their systems effectively. For the original specifics, I'll talk with the sourceforge people about an appropriate exim config for their problem. Nigel. -- [ - Opinions expressed are personal and may not be shared by VData - ] [ Nigel Metheringham Nigel.Metheringham(a)VData.co.uk ] [ Phone: +44 1423 850000 Fax +44 1423 858866 ]

1 0

Re: [Mailman-Developers] Re: [Mailman-Users] Problem with qrunner and too much incoming mail
by J C Lawrence Nov. 6, 2000

Nov. 6, 2000

On Sat, 4 Nov 2000 10:29:15 -0800 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: >> > Once a connetion to the MX is established, bulkmail would then >>> just start delivering messages to it until the bin was emptied. >>> Any i/o blocks in any of the processes will allow async* to >>> switch to a different delivery channel. We may need to do some >>> explicit channel management to make sure some are not starved. >> Ouch. I really don't like this idea. > Neither do I. This is actually something that I've looked at long > and hard in my non-mailman server work. After a fair amount of > work and research, I finally came to the conclusion that you are > MUCH better off letting the MTA do the MTA's work, and letting the > MLM do the MLM work, and once you make the decision that the MLM > has to *also* become an MTA, you're doing down a road you don't > want to travel. Agreed. There's also the simple aspect of the fact that optimising for the very large case (which needs the MLM-sepcific MTA supports ala ListServ) is near on pessimal for the small and medium cases. Spiffy domain based routing and near-target explosion just don't make a whole lot of sense for lists with a few hundred/thousand members. Asides from which, the optimisations for mail delivery to a massive number of targets and mass mail delivery to a comparitively constrained target set can be rather different. That's ugly territorry. We don't need to go there yet. > Instead of reinventing the MTA wheel, I think we're much better > off coming up with an MTA -> MLM interface that's very flexible > and highly configurable (most especially in how to deliver and how > much to parallelize the infeed to the MLM), and then focus on how > to tune the MTA and MLM through documentation. Agreed. There's not a whole lot of tuning you can do on SMTP hand-offs to the MTA (which is necessary as a feature to support non-localhost MTAs which is in turn needed by many sites), but some MTAs have divers command line options that can be useful in an MLM delivery world (peeking quickly at the Exim docs). > Splitting the inbound and outbound queue would be my first thing > here, and probably split bounces into a third queue. That's a > pretty quick, easy optimization that makes sure the end user sees > fast response without being bogged down by deliveries, and that's > a huge perception issue. Then focus on parallelizing the delivery > from mailman into the MTA, and make that configurable so each > admin can tune it to their system and needs. <nod> Without spending any time collecting metrics, it does look like the low hanging fruit is in handling the MLM->MTA handoff. Given that the distribution of posts across lists is often uneven (and unpredictable), and that the optimal parallelism (N) is fairly fixed for a given machine, any pattern which supported keeping N pipes stuffed throwing things at the MTA (with minimal locking) would seem fair. As for splitting the queues, yep, hadn't thought about that aspect, but that's an obvious contention area for an active list. >> As discussed previously amongst Chuq, Nigel and I, the needs of >> large list server systems are rather different from the normal >> home hobbyest requirements, but are not compleatly alien. >> However, the needs of very large list installations (cf ListServ, >> Egroups, or SourceForge) are rather different yet again. > This is a basic reality -- things don't scale. Or worse, they > scale for a while, and then you need to switch paradigms. I found > that one out the hard way. If someone wants a rhetoric on how to > scale mail list servers infinitely, I'd be happy to explain how, > since I've had to develop an architecture to do so. the nice thing > is, it can be done without exceptional engineering hassles -- but > it's not just adding another daemon or a faster CPU (although > those are solutions for parts of it, just not ALKWAYs the > solutions) We should really have lunch some time. I'm currently working down off San Tomas and Saratoga (startup, oddly enough next door to BeOpen). I'm wrapping up this contract this month and looking for next one, so time is a little tight here, but I'd love to get together for a chat. You over at the main Apple campus? >> I'm not convinced of the value in beating on Mailman to support >> the (comparitively rare if high profile) very large installations >> when the current (much larger and more common Mailman-wise) >> mid-size realm still needs attention. Certainly, such changes >> should not detract from Mailman's current level of suitability >> for smaller installations. > I think we can build a Mailman that does this, at least for, oh, > 95% of the universe out there, and the other 5% are going to have > custom solutions anyway (or should!). What we don't want to do is > screw up Mailman for the "typical" user to make it work for the > big site; but we also don't want Mailman to get a reputation as a > "small server only" system, because it'll cause people to reject > it in implementations. Fortunately, I don't think you need to do > that. It just needs some tweaking. <nod> We must stop agreeing like this. Its getting embarassing. > And for sites this *doesn't* solve, it's either because they're > doing the 5 pounds in a 1 pound bag thing, or they probably need > to start hiring people like us to custom build someething. Ungghhhh. -- J C Lawrence Home: claw(a)kanga.nu ---------(*) Other: coder(a)kanga.nu http://www.kanga.nu/~claw/ Keys etc: finger claw(a)kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Users] Problem with qrunner and too much incoming mail
by J C Lawrence Nov. 4, 2000

Nov. 4, 2000

<<This was originally off-list, moved back on list with mutual permission>> On Fri, 3 Nov 2000 17:01:01 -0800 Marc MERLIN <marc(a)merlins.org> wrote: > On Fri, Nov 03, 2000 at 04:56:33PM -0800, J C Lawrence wrote: >> Yup, that's what I mean by pass-thru. > Ok, I wasn't sure. No worries. >> of having Exim do that a loooong time back but I don't know if >> anything ever happened there (there are obvious timeout and >> performance issues when delivering to slow/unresponsive sites >> (think about behavioural changes mid-negotiation)). > I see. So the message gets spooled as a backup and erased right > afterwards if the current process was able to deliver it I > suppose. Quite. Note that this leaves with the same disk IO that not doing pass-thru does. The only time pass-thru gains you anything is when you can hand the message off very close to as fast as, or faster than you can receive it. In many cases I could see this being a gain given free queue runners and minimal queue runner negotiation overhead or complexity. >> To deal with the physical IO problems, the very large volume mail >> server companies I've dealt with and talked to (eg my last >> client, Critical Path) use solid state disks for their mail >> queues (in CP's case under a modified QMail, soon to be replaced >> by NPLex). Its a bloody expensive solution to be sure, but its >> also a fast one. > It can come down to that, but that's throwing money at hardware > instead of fixing mailman's deficiencies. True. Mailman is the real bottleneck here. A simple, and computationally and code-wise cheap solution would be to split the queues per list. Simply have each list have a private outbound queue area (this may be done already, I haven't looked at v2 in detail and am short of time right now). Then rather than threading, simply have per-list queue locks. The qrunner then forks N children (configurable) which proceed to attempt to empty each queue in order, each one locking its queue as it gets it (such that the others pass over it) etc. Thus deliveries to the MTA are parallelised. Its a fairly light weight code change (private queues and a child forking qrunner) but it skips all the overhead of threading and fancy locking schemse. Further, as the number of posts per list queue is likely to be low, the iteration rate, and therefore the lock period (for other posts to join a given list's queue will be small. If Mailman v2 currently uses private queue (dunno) then your likely problem is twofold: the thrash rate of walking 10K private queues plus the MTA receipt problem. This problem is mostly unique to the very large number of lists your run. A possibly better tack there (given a reasonbly high frequency qrunner polling rate) would probably be to store the queue entries in hash trees (hash by list or hash by post) and then go from there. Adjust the hash for say N buckets (where N is the total number of parallel qrunners you want). This has the advantage of minimising disk and directory thrashing. The disadvantage is that the code impact would be correspondingly large. > On Fri, Nov 03, 2000 at 04:57:11PM -0800, J C Lawrence wrote: >> > Yeah, that's just what I was saying here. ...etc >> >> BTW: Did you inbtend this to be off list? > Yeah, it's not mailman related, it's exim stuff, thus off topic > :-) <nod> This conversely whould probably go back to the list. Mind if I cross it? -- J C Lawrence Home: claw(a)kanga.nu ---------(*) Other: coder(a)kanga.nu http://www.kanga.nu/~claw/ Keys etc: finger claw(a)kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

1 0

Fix (I think) for duplicates problem
by barry＠wooz.org Nov. 3, 2000

Nov. 3, 2000

The last followup to bug #117015 by deejster provided what I think is the critical clue to the strange duplication problem. Given that recipe I was indeed able to reproduce the bug: - a message gets held for whatever reason - the message is approved, but there are smtp errors during delivery - the message gets re-queued, but with a file name composed of the old filebase, not the new filebase. - fwap! you've now got a duplicate of the original message Here's a fix for this problem. It removes the filebase value from the original msgdata, so that this will be recalculated when the file is re-queued after approval. For those of you seeing duplicates, please apply this patch and let me know if it fixes your dup problem. This is important enough to spin a second release candidate. -Barry Index: ListAdmin.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/ListAdmin.py,v retrieving revision 1.46 diff -u -r1.46 ListAdmin.py --- ListAdmin.py 2000/10/10 06:33:31 1.46 +++ ListAdmin.py 2000/11/02 23:05:57 @@ -208,6 +208,12 @@ return LOST msg = Message.Message(fp) msgdata['approved'] = 1 + # Calculate a new filebase for the approved message, otherwise + # delivery errors will cause duplicates. + try: + del msgdata['filebase'] + except KeyError: + pass # Queue the file for delivery by qrunner. Trying to deliver the # message directly here can lead to a huge delay in web # turnaround.

2 2

Get listed. :)
by barry＠wooz.org Nov. 1, 2000

Nov. 1, 2000

I'm putting together the on-line documentation for the 2.0 release and I'd like to include a list of sites that run Mailman. I've got Python.Org and lists.apple.com (if that's cool with you Chuq ;). Please send any others you'd like to include to me directly. Thanks, -Barry

4 3

mailman-developers problem.
by Chuq Von Rospach Nov. 1, 2000

Nov. 1, 2000

Barry, we have a problem with mailman-developers. I responded to your mail, and my response went to -users and -developers, and I got THREE copies. I got one from -users. I got one from -developers. and then I got a third copy of the -users mail, resent to -developers. looks like someone on the list is doing something stupid: To: catholic.org(a)catholicnet.org.uk Received: from dinsdale.python.org (dinsdale.cnri.reston.va.us [132.151.1.21]) by webmail6.catholic.org (8.10.1/8.10.1) with SMTP id eA14OAQ18572 for <pdj(a)catholic.org>; Wed, 1 Nov 2000 04:24:11 GMT Received: from dinsdale.python.org (localhost [127.0.0.1]) by dinsdale.python.org (Postfix) with ESMTP id 0C2981CE17; Tue, 31 Oct 2000 23:24:11 -0500 (EST) the third copy is routing through that address. Can you please shoot this person? the catholic.org address seems to be hosed. -- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com) Be just, and fear not.

2 2

[Mailman-Users] forwarded message from Guido van Rossum (fwd)
by Thomas Wouters Nov. 1, 2000

Nov. 1, 2000

Anyone else notice something weird about the mail below ? It was sent by Guido to python-dev, where Barry picked it up from and sent it to both mailman-developers and mailman-users. I am only subscribed to mailman-developers, and I got it twice. What's more, the second mail seems to have gone to mailman-developers *through* mailman-users. Notice the extra list-sig at the obttom of the mail, and extra [listname] in the subject. Bug in Mailman ? ;-P ----- Forwarded message from "Barry A. Warsaw" <barry(a)wooz.org> ----- Date: Mon, 30 Oct 2000 11:24:57 -0500 (EST) From: barry(a)wooz.org (Barry A. Warsaw) Sender: mailman-developers-admin(a)python.org To: catholic.org(a)catholicnet.org.uk Cc: mailman-developers(a)python.org Cc: mailman-users(a)python.org Subject: [Mailman-Developers] [Mailman-Users] forwarded message from Guido van Rossum Date: Fri, 27 Oct 2000 20:42:42 -0500 From: Guido van Rossum <guido(a)python.org> Sender: python-dev-admin(a)python.org To: python-list(a)python.org (Python mailing list), python-announce-list(a)python.org, python-dev(a)python.org Subject: [Python-Dev] PythonLabs Team Moves to Digital Creations To all Python users and developers: [[ .. snip .. ]] _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://www.python.org/mailman/listinfo/python-dev ------------------------------------------------------ Mailman-Users maillist - Mailman-Users(a)python.org http://www.python.org/mailman/listinfo/mailman-users _______________________________________________ Mailman-Developers mailing list Mailman-Developers(a)python.org http://www.python.org/mailman/listinfo/mailman-developers ----- End forwarded message ----- -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

2 1

Re: [Mailman-Developers] My admins speak out...
by Dan Mick Nov. 1, 2000

Nov. 1, 2000

> if getattr(mlist, 'auto_reject', 0): > raise DiscardMessage Very nice; had no idea that one little exception is all it takes. Is there anything similarly-simple to generate a generic bounce message and discard, or is that all "call some reply method and then raise DiscardMessage"?

2 1