Mailman 3 May 2000 - Mailman-Developers

Re: [Mailman-Developers] Huge lists
by J C Lawrence 26 May '00

26 May '00

On Thu, 25 May 2000 11:05:57 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > I was thinking last night that what would REALLY, REALLY be useful > here is an extended SMTP protocol that allows the VERPing to be > introduced by the receiving SMTP server, rather than the delivery > server or MLM. And after thinking about it, I went and laid down > in a dark room until I got over it... (snicker). But if you think > about it, the downside to VERP is you lose the efficiency of > batching multiple addresses into a single transaction, so the > solution is to extend SMTP to allow us to maintain that effeciency > while building in the VERPing data at time of delivery... It would be interesting to get this to work across .forwards and other things that rewrite the envelope. -- J C Lawrence Internet: claw(a)kanga.nu ----------(*) Internet: coder(a)kanga.nu ...Honorary Member of Clan McFud -- Teamer's Avenging Monolith...

1 0

Re: [Mailman-Developers] Huge lists
by J C Lawrence 26 May '00

26 May '00

On Thu, 25 May 2000 10:32:41 +0100 Nigel Metheringham <Nigel.Metheringham(a)VData.co.uk> wrote: > Wietse had some figures on MTA performance analysis which he used > as part of the design process for Postfix. He concluded that disk > I/O was *the* limiting factor for an MTA... Excellent point. > If we have a million user list... and a message of a few K, I'm > not sure I want to have a few GB of queue space taken up. If some > idiot sends a 1M attachment I doubt many of us have the TB spool > space. FWLIW, this is the reason I specced having the handler process feed the MTA at a set rate, thus allowing the MTA to attempt to drain the spool as it comes in. Without some sort of feedback (perhaps `df`?) you can't guarantee anything of course, but it *should* help without getting complex of MTA specific. > Having said that I *really* would like the possibility of the > occaisional message (maybe even just the password > reminders.. although I'd prefer a method where some messages if > the list was in a state where it has recently seen bounces that it > cannot tie to a particular subscriber) be sent out using VERP. > However then we also need to recode the MTA incoming handling to > take that - aliases don't cut it any more. Not exactly. I did exactly this for one of my list servers, tho not exactly in the VERP style. What I did was to to have every N'th message broadcast be pseudo-VERPed. I did this by encoding special subscriber-specific headers into the subscriber-individual messages, and then looking for those headers in bounces (I haven't found a single MTA whoses bounces don't at least quote the original message's headers). Spot the special headers, flag the relevant account, and so forth. Very clean. Very effective. Almost all the benfits of VERP without the pain (I don't consider the per-subscriber subscribe/unsubscribe addressing etc interesting). > chuqui(a)plaidworks.com said: >> There are exchange sites out there who's idea of a bounce message >> is to return the mail to the "to:" line with only the Message-ID >> changed. you can imagine how much fun THAT is. ... > That particular one you mention should be blocked from the net - > presumably their upstream is clueless too. I noticed a while back that my Exim reject logs were full of entries for a system called "<something>_mail" (forget what "Something" was). Underscores are of course not welcome for HELOs, so Exim was rejecting the traffic. To cut to the chase, it turned out to be an Exchange box at a company that was trying to get me to do some consulting for them, and who were very upset that I wasn't replying to their email. Further, it turned out, or so they said, that the MS manual for Exchange strongly recommended that they name the box "<something>_mail" Sometimes I wish more Exchange sites would read and follow their manuals. -- J C Lawrence Internet: claw(a)kanga.nu ----------(*) Internet: coder(a)kanga.nu ...Honorary Member of Clan McFud -- Teamer's Avenging Monolith...

1 0

Re: [Mailman-Developers] Huge lists
by Nigel Metheringham 25 May '00

25 May '00

[Personal Ccs deleted... list only this time] Its nice to see you folks have been enjoying yourselves whilst I sleep. However I now have the advantage, so will respond to the dozen or so messages in the last batch :-) ] chuqui(a)plaidworks.com said: > throwing hardware at a problem isn't always possible. but the place > where rolling your own internal MTA starts becoming useful is when > the list is big enough that the disk I/O involving the MTA starts > becoming the significant limiter. With sendmail 8.9.x, that's fairly > easy to run into. With sendmail 8.10, it seems to be better, and the > multiple queue stuff solves a multitude of problems involving huge > directory structures. Wietse had some figures on MTA performance analysis which he used as part of the design process for Postfix. He concluded that disk I/O was *the* limiting factor for an MTA - remember that to comply with the RFCs you have to commit incoming data to stable storage before acknowledging receipt (ie the positive reply to SMTP end of data) - in all current mainstream MTAs that means that the queue file has to be closed and synced. Pushing data down to the rust and ensuring its there stably limits things drastically. Wietse's tests should be on www.postfix.org > VERP exacerbates the problem, since # of batches sent to the MTA > equals the # of addresses, which explodes the number of control > files, which... So at some point, it makes sense to deliver direct to > recipient rather than build batches into the MTA, and completely > avoid the disk I/O and deliver right out of the database to the > receiving SMTP client. You could strongly parallelize the delivery > setup because you'd do away with all of the MTA overhead, and do all > sorts of fun things, like prioritize your delivery sorting and the > like. If we have a million user list... and a message of a few K, I'm not sure I want to have a few GB of queue space taken up. If some idiot sends a 1M attachment I doubt many of us have the TB spool space. Having said that I *really* would like the possibility of the occaisional message (maybe even just the password reminders.. although I'd prefer a method where some messages if the list was in a state where it has recently seen bounces that it cannot tie to a particular subscriber) be sent out using VERP. However then we also need to recode the MTA incoming handling to take that - aliases don't cut it any more. ------ The queueing stuff is interesting, although big list focused boxes are likely to not be the primary users of mailman - however if the exim list is anything to go by those (big list) users will be among the most vocal and contribute most ideas and code. [I have worked on big mail systems, but not really big list systems] claw(a)kanga.nu said: > Sorting the RCPT TO list by domain costs us very little (esp if we > sort on insertion), and can help users of dumb MTAs considerably. Yup... chuqui(a)plaidworks.com said: > You could make a good argument that the best way to optimize is to > create one mail batch per unique hostname, up to SMTP-MAX-RCPTS, at > which point you split it into num_addrs/SMTP-MAX-RCPTS batches for > that hostname, and then let the MTA sort if out from there. Counter examples are always problems.... The biggest UK ISP group (several "virtual" ISPs use the same bulk ISP service set) has a few million users each of whom have their own domain name - so you will find that *.freeserve.co.uk (around 2 million domains) all goes to the same batch of MXes. This means that a good approach (for this type of account naming) would be to pack in sets of addresses in reverse domain order until you had a batch of SMTP-MAX-RCPTS (obviously you additionally optomise this by also making sure that a single domain is not split over 2 batches unless the number of addresses in that domain are larger than a batch). As for a quick description of exim queueing practices:- - Queues are processed in a basically random order... incoming messages however *normally* have a delivery process invoked for them immediately after end-of-smtp-data (there is policy associated here - can be tweaked) - Each domain/address/message have retry hints associated with it if the retry time for a message/domain/address has not been hit then it is not taken further - so often a group of messages in the queue are skipped on each queue run because their retry time has not arrived - Exim resolves all undelivered addresses in a message and groups them by MX (lets ignore alternative delivery schemes here) - Each MX set has delivery attempted (there may be parallelism here) - If the MX set can be contacted then the message is shoved down the pipe, then the hints database is checked for other messages outstanding on that MX set - if so then the pipe is passed to another delivery process invoked on one of the waiting messages - If MX set was *not* successful then the hints are updated to say this message has addresses outstanding on that MX So in the normal case each delivery process delivers only to the addresses in the message its dealing with - each message is independent so you may have several SMTPs to the same place for different messages. If things clog up then hints help make things more efficient. [these are hints - sometimes they are ignored, and trashing the hints db is quite OK]. This all works pretty well in practice. You can if you want a particular type of efficiency rearrange things - ie make all messages resolve, but only deliver on queue runs, which means that messages for the same destination host are nearly always batched down a single SMTP session. [On per-MTA documentation] Lets start bullying^Wpersuading people to contribute some documentation on this stuff or pointers to existing MTA documentation that addresses this. The question of MTA configuration for medium size lists is pretty common, so there must be tuning data around. I guess I could collate if needed [sigh] Big lists are a different issue - you need to *choose* your MTA and hardware within your constraints for that. Tuning is probably a consultancy job for those. chuqui(a)plaidworks.com said: > There are exchange sites out there who's idea of a bounce message is > to return the mail to the "to:" line with only the Message-ID > changed. you can imagine how much fun THAT is. More special bounce filters needed :-) I *like* the way that mailman is now dealing with an impressive proportion of bounces. I need to write an extra filter to make it drop delay warning messages, other than that theres very little stuff getting through to me in the way of bounces. That particular one you mention should be blocked from the net - presumably their upstream is clueless too. Nigel. -- [ - Opinions expressed are personal and may not be shared by VData - ] [ Nigel Metheringham Nigel.Metheringham(a)VData.co.uk ] [ Phone: +44 1423 850000 Fax +44 1423 858866 ]

2 1

(no subject)
by Ken Manheimer 25 May '00

25 May '00

Re the smart archives interest, one of the guys (newly) here has the interest and may have the mandate to be doing something significant, of course using zope rather than php/mysql or midgard. (Newcomers to the mailman community may see some signs of preference for python-based implementations of things in the mailman community - read, python bigotry - and favoring of zope *could* be seen as another symptom of that. While favoring zope is really a lot more than just a python preference, digital creations just recently announced a rather exciting project, with partnership of SourceGear, for implementation of perl based methods for zope, so people articulate and extend zope using perl. Just incidentally.-) Smart archives for mailman (or for any mailling list collaboration) has been a dear wish of mine since i was working on mailman. It was heightened by joining digital creations, but i got distracted learning zope and going in some other directions, and never got to the zope/mailman integration. Though i've gotten pretty far from mailman development, i'm still extremely eager to see smart archives happen - i think the collaborative usefulness of mailing lists is severely limited without a comprehensive, well integrated smart archive. I know barry and others have been similarly psyched to see such a thing - i hope ethan's mandate and interest does head in this direction, we'll see. If se, there may be some good opportunities for synergy with the other mailman developers along the way... Ken Manheimer klm(a)digicool.com On Thu, 25 May 2000 00:05:38 -0700, Nigel Metheringham <Nigel.Metheringham(a)VData.co.uk> wrote: > On Wed, 24 May 2000 22:13:08 -0700 Chuq Von Rospach > <chuqui(a)plaidworks.com> wrote: > > [lots of stuff] > > > I used to use it, and then switched my web archives to a full > > forum system (web crossing) and crosslinked everything. that has > > its advantages and disadvantages. > > One of my list members has been advocating WebCrossing. What do you > think of it? It seemed excessively constraining to me, especially > since I'm heading toward a massively WikiWiki-fied setup (every page > can be commented on Wiki-style, all comments are free-standing Wiki > entities etc etc etc). > > >> -- Supports archive searching by MessageID. I've an MTA hack > >> that inserts a MessageID-based URL into all outgoing Mailman list > >> traffic so the user can just hit the URL and be taken to that > >> message in the archives (searches the MHonArc DB, useful for > >> thread reference etc). > > > Interesting hack. Very interesting hack. > > <bow> Wish it were original to me. One of my list members came up > with the idea and then went and implemented it. Its somewhere in > Keystone under Tasks... > > > you could do something really nice with PHP and MySQL, too, and do > > away with MHonarc, and parse/templatize the text on the > > fly. that's sort of where I'm headed down the road.... > > Yeah, I've thought about that but I really just don't see enough > advantage to justify the time it would take to get something better > than I have now. Eric Hood (MHonArc author) has been also > threatening to do something here for ages. > > Its awfully tempting tho just on a "cool!" factor. In response, on Thu, 25 May 2000 00:29:48 -0700, chuqui wrote: > >One of my list members has been advocating WebCrossing. What do you > >think of it? > > Not appropriate for this list. Let's talk offline. I'm designing it > out of my systems in favor of other things, but the reasons are > complex -- and I've recommended it INTO at least one major > development project at the same time. So I guess the answer is "it > depends, but I'm not going to be using it myself..." > > > > you could do something really nice with PHP and MySQL, too, and > > > >Yeah, I've thought about that but I really just don't see enough > >advantage to justify the time it would take to get something better > >than I have now. > > I wonder how much of this could be driven out of something like > Midgard? But loading your entire archives into a database gives you > the ability to do all sorts of interesting linking and searching and > stuff, and "all" you'd need is some email->XML converter, and then... > > Oh, man. We need to at least pretend to be on topic for this list, > but I need a white board and a pen... (scribbly scribble...)

1 0

Re: [Mailman-Developers] Huge lists
by J C Lawrence 25 May '00

25 May '00

On Wed, 24 May 2000 22:13:08 -0700 Chuq Von Rospach <chuqui(a)plaidworks.com> wrote: > At 9:52 PM -0700 5/24/2000, J C Lawrence wrote: >> Umm, true. Looking at it again, and doing a quick check of my >> user base's MXing, I suspect we're dealing with a less than 1% >> gain. Bigger fish are available. Methinks my brain was farting. > Nope, just getting a bit ahead and thinking of something that's > fun and technically challenging. Been there, done that... (grin). Well, yes. >> I don't believe that a list server has any business handling MX >> sorting unless it is also taking responsibility for being the >> list MTA. As Mailman isn't, its a moot point. > And that's an issue I've been wrestling with a lot -- do I do a > specialized MTA? Or do I let the MTA do its job. After going back > and forth on this for weeks, given my current delivery rates, I've > decided to let the MTA do its job, and wait on writing a > specialized MTA until I need that last couple of percent of > performance. Moving to 8.10.1 seems like an easy performance bump, > postfix looks like it'll buy me even more, and so while doing all > the MXing and stuff would be fun, it can wait. I'm a big believer in being lazy. I don't like handling problems that other people seem to have done a decent enough job already unless they're *really* interesting. There are already enough interesting things out there to do to keep me busy, so that doesn't happen often. Ergo, if the MTA looks like it can do it, can be persuaded to do it, and that means I really don't have to, then well, who am I to argue? QMail, with which I'm becoming uncomfortably familiar, is bloody fast. I hope/expect that Postfix is very similar. Its tough to image a situation where my time and effort in replacing them (as a solo effort) would actually be worth it as versus throwing hardware at the problem or chatting up Wietse & co. I've written list servers and mini-MTAs before. There's a fair bit of hidden complexity and brain hurt in there I don't mind avoiding. > queue management is another issue. that's one place majordomo is > weak at, because it doesn't. Everything is delivered as it comes > in, so bursts can take a system to its knees. True. Were Mailman asynchronous, a pattern as below would seem useful: There is never more a single "queued message handler" process (maybe multi-threaded, or not). That process guarantees not to feed messages to the MTA any faster than XXX messages per second/minute, and to stop such feeding were system load to rise above ZZZ. The single instance rule prevents multiple handler processes for multiple mailing lists maxxing out the MTA as they all dump simultaneously. The problem of multiple list servers (boxes) dumping simultaneously to a remote MTA is properly, I believe, outside of Mailman's purview. I don't see a value in trying to monitor MTA queue size. Too MTA specific. Monitoring list server queue size and implementing fairness algorithms in emptying the queue across multiple lists is worth looking at tho (arguably round-robin is "good enough" and is very simple to do, but it ain't pretty when things get ugly). > Another thing to worry about... On my big system, I only do a few > mailings a week, but they bunch together. So I've had to do a > bunch of work on making sure the system deals with this > rationally... when we were doing one mailing on a given day, that > was easy, but we're doing both a text and an HTML variant going > out together, and that really complicates life. Yeah, volume, as versus number of messages, can be another problem if only for disk IO. > Well, this is probably preaching to the choir, but I've gotten > quite convinced that you isolate every piece you can from every > other piece, and document the interfaces. that makes it quite easy > to swap out a new piece without affecting the rest of the system ... This is often called, "programming by contract". Its a Good Thing. >> -- Allows archived messages to be replied to on the web via the >> archive page (replies post to the list). > Nice! does it restrict posting access to registered users or is it > open? I let the list server handle that. As I insert a special header in messages coming from the web interface its very easy to configure Mailman to hold such posting for moderator approval (privacy options page) > I used to use it, and then switched my web archives to a full > forum system (web crossing) and crosslinked everything. that has > its advantages and disadvantages. One of my list members has been advocating WebCrossing. What do you think of it? It seemed excessively constraining to me, especially since I'm heading toward a massively WikiWiki-fied setup (every page can be commented on Wiki-style, all comments are free-standing Wiki entities etc etc etc). >> -- Supports archive searching by MessageID. I've an MTA hack >> that inserts a MessageID-based URL into all outgoing Mailman list >> traffic so the user can just hit the URL and be taken to that >> message in the archives (searches the MHonArc DB, useful for >> thread reference etc). > Interesting hack. Very interesting hack. <bow> Wish it were original to me. One of my list members came up with the idea and then went and implemented it. Its somewhere in Keystone under Tasks... > you could do something really nice with PHP and MySQL, too, and do > away with MHonarc, and parse/templatize the text on the > fly. that's sort of where I'm headed down the road.... Yeah, I've thought about that but I really just don't see enough advantage to justify the time it would take to get something better than I have now. Eric Hood (MHonArc author) has been also threatening to do something here for ages. Its awfully tempting tho just on a "cool!" factor. -- J C Lawrence Home: claw(a)kanga.nu ----------(*) Other: coder(a)kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] Huge lists
by J C Lawrence 25 May '00

25 May '00

On Wed, 24 May 2000 16:18:10 +0100 Nigel Metheringham <Nigel.Metheringham(a)VData.co.uk> wrote: > However we're getting to religious arguments here. Suffice it to > say that should mailman go VERP only, I move to something else :-) Given how MTA agnostic Mailman is, and how poor VERP support is under several popular MTAs, I'd rate this as unlikely. -- J C Lawrence Home: claw(a)kanga.nu ----------(*) Other: coder(a)kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

1 0

Re: [Mailman-Developers] Huge lists
by J C Lawrence 25 May '00

25 May '00

On Wed, 24 May 2000 15:52:52 +0100 Nigel Metheringham <Nigel.Metheringham(a)VData.co.uk> wrote: > claw(a)kanga.nu said: >> Actually on the input side it would be 2M/SMTP_MAX_RCPTS where >> SMTP_MAX_RCPTS defaults to 500. Additionally, Exim (I haven't >> checked Postfix, but I doubt it) does not collapse identical >> spool entries with different RCPT TO lists, so you're still left >> with the same number (checking for identicallity is expensive). > You pedant :-) Why, yes! > Granted although that raises the other issue... Barry mentioned > yesterday that he was putting the dropped SMTP_MAX_RCPTS > functionality back in... but I thought it had deliberately gone on > the basis that mailman was giving the mail delivery job to the MTA > and not trying to second guess it. SMTP_MAX_RCPTS needs to be there as some MTAs have internal limits on the number of RCPT TOs. It used to be the old BitNet and big iron machines that were most known for this. Don't know wha't out there now, tho I remember getting a couple bounces for that reason a couple years back. > If we do split the list, then ideally it should be sorted.... at > least so identical domains go together and maybe more > intelligently than that.... and we're back to trying to second > guess the MTA at its own job. Sorting the RCPT TO list by domain costs us very little (esp if we sort on insertion), and can help users of dumb MTAs considerably. Its seems a *reasonable* tradeoff, tho argument could also be made that it merely supports the continued use of broken software -- but I'll leave that argument for another time. Yep, its second guessing the MTA, but its a cheap, cost effective, minimal impact guess that has nearly NO punitive effect on mailman itself. >> I would expect them to have configured their MTA to not attempt >> pre-lookups for messages arriving from MailMan. > I guess that we need a per MTA tuning/configuration document. Aaaargh. Yes. > I pretty much have that in the exim mailman howto... BTW, well done on the Exim HOW-TO. Nicely done. > (always willing to take contributions/comments on that BTW - > especially if they suggest solutions rather than just raising > criticisms). Without going and re-reading it, about the only thing I can think to add to it would be turning off domain checking for localhost RCPTs as per our recent comments if that's not there already. BTW: Does Exim index its spool on the basis of MX? It doesn't *look* like it per watching it, but I haven't checked. How about Postfix? Anybody know? > Have we got that data for the other MTAs? Nope. User written of course, and well, you know. > Of course we could have comparisons of the suitabilities of > different MTAs if people enjoy a decent flamefest. Of course not. Everybody knows that Microsoft Exchange is the one true MTA and all else are but pale imitations. -- J C Lawrence Home: claw(a)kanga.nu ----------(*) Other: coder(a)kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

2 1

Re: [Mailman-Developers] Huge lists
by Nigel Metheringham 24 May '00

24 May '00

chuqui(a)plaidworks.com said: > If you VERP, so you have one piece of email per subscriber per > message, you stop trying to outguess the MTA, since it can look at > everything and decide a delivery strategy Yes, but you have done this by denying the MTA the chance to optomise delivery by packing multiple similar recipients together. The MTAs delivery strategy is indeed simpler.... as in we don't need no stinking optomisation - it has the freedom to choose... from the one option you have left it. However we're getting to religious arguments here. Suffice it to say that should mailman go VERP only, I move to something else :-) Nigel. -- [ - Opinions expressed are personal and may not be shared by VData - ] [ Nigel Metheringham Nigel.Metheringham(a)VData.co.uk ] [ Phone: +44 1423 850000 Fax +44 1423 858866 ]

2 1

Re: [Mailman-Developers] Huge lists
by Nigel Metheringham 24 May '00

24 May '00

claw(a)kanga.nu said: > Actually on the input side it would be 2M/SMTP_MAX_RCPTS where > SMTP_MAX_RCPTS defaults to 500. Additionally, Exim (I haven't checked > Postfix, but I doubt it) does not collapse identical spool entries > with different RCPT TO lists, so you're still left with the same > number (checking for identicallity is expensive). You pedant :-) Granted although that raises the other issue... Barry mentioned yesterday that he was putting the dropped SMTP_MAX_RCPTS functionality back in... but I thought it had deliberately gone on the basis that mailman was giving the mail delivery job to the MTA and not trying to second guess it. If we do split the list, then ideally it should be sorted.... at least so identical domains go together and maybe more intelligently than that.... and we're back to trying to second guess the MTA at its own job. > I would expect them to have configured their MTA to not attempt > pre-lookups for messages arriving from MailMan. I guess that we need a per MTA tuning/configuration document. I pretty much have that in the exim mailman howto (always willing to take contributions/comments on that BTW - especially if they suggest solutions rather than just raising criticisms). Have we got that data for the other MTAs? Of course we could have comparisons of the suitabilities of different MTAs if people enjoy a decent flamefest. Nigel. -- [ - Opinions expressed are personal and may not be shared by VData - ] [ Nigel Metheringham Nigel.Metheringham(a)VData.co.uk ] [ Phone: +44 1423 850000 Fax +44 1423 858866 ]

2 1

Re: [Mailman-Developers] Huge lists
by J C Lawrence 24 May '00

24 May '00

On Wed, 24 May 2000 09:51:29 +0100 Nigel Metheringham <Nigel.Metheringham(a)VData.co.uk> wrote: > Remember that unless the sender side is using qmail or VERP then 2 > million addresses all in this domain would normally optomise down > to a single SMTP message transaction with 2 million RCPT TO's. Actually on the input side it would be 2M/SMTP_MAX_RCPTS where SMTP_MAX_RCPTS defaults to 500. Additionally, Exim (I haven't checked Postfix, but I doubt it) does not collapse identical spool entries with different RCPT TO lists, so you're still left with the same number (checking for identicallity is expensive). > You may also find that you get an unexpected speed up at the > inject end since the domain name may (certainly should) be cached > in the MTA and/or local dns server/cache and so if the MTA does > verification/routing during SMTP receipt the first address would > prime the config and all the others would shoot through very > quickly. True. However for anybody running such a large mailing list, I would expect them to have configured their MTA to not attempt pre-lookups for messages arriving from MailMan. Certainly that's how I've configured mine. -- J C Lawrence Home: claw(a)kanga.nu ----------(*) Other: coder(a)kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

2 1