Re: [Mailman-Developers] Huge lists
On Wed, 24 May 2000 15:52:52 +0100 Nigel Metheringham <Nigel.Metheringham@VData.co.uk> wrote:
claw@kanga.nu said:
Actually on the input side it would be 2M/SMTP_MAX_RCPTS where SMTP_MAX_RCPTS defaults to 500. Additionally, Exim (I haven't checked Postfix, but I doubt it) does not collapse identical spool entries with different RCPT TO lists, so you're still left with the same number (checking for identicallity is expensive).
You pedant :-)
Why, yes!
Granted although that raises the other issue... Barry mentioned yesterday that he was putting the dropped SMTP_MAX_RCPTS functionality back in... but I thought it had deliberately gone on the basis that mailman was giving the mail delivery job to the MTA and not trying to second guess it.
SMTP_MAX_RCPTS needs to be there as some MTAs have internal limits on the number of RCPT TOs. It used to be the old BitNet and big iron machines that were most known for this. Don't know wha't out there now, tho I remember getting a couple bounces for that reason a couple years back.
If we do split the list, then ideally it should be sorted.... at least so identical domains go together and maybe more intelligently than that.... and we're back to trying to second guess the MTA at its own job.
Sorting the RCPT TO list by domain costs us very little (esp if we sort on insertion), and can help users of dumb MTAs considerably. Its seems a *reasonable* tradeoff, tho argument could also be made that it merely supports the continued use of broken software -- but I'll leave that argument for another time.
Yep, its second guessing the MTA, but its a cheap, cost effective, minimal impact guess that has nearly NO punitive effect on mailman itself.
I would expect them to have configured their MTA to not attempt pre-lookups for messages arriving from MailMan.
I guess that we need a per MTA tuning/configuration document.
Aaaargh. Yes.
I pretty much have that in the exim mailman howto...
BTW, well done on the Exim HOW-TO. Nicely done.
(always willing to take contributions/comments on that BTW - especially if they suggest solutions rather than just raising criticisms).
Without going and re-reading it, about the only thing I can think to add to it would be turning off domain checking for localhost RCPTs as per our recent comments if that's not there already.
BTW: Does Exim index its spool on the basis of MX? It doesn't *look* like it per watching it, but I haven't checked.
How about Postfix? Anybody know?
Have we got that data for the other MTAs?
Nope. User written of course, and well, you know.
Of course we could have comparisons of the suitabilities of different MTAs if people enjoy a decent flamefest.
Of course not. Everybody knows that Microsoft Exchange is the one true MTA and all else are but pale imitations.
-- J C Lawrence Home: claw@kanga.nu ----------(*) Other: coder@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--
At 3:41 PM -0700 5/24/2000, J C Lawrence wrote:
SMTP_MAX_RCPTS needs to be there as some MTAs have internal limits on the number of RCPT TOs. It used to be the old BitNet and big iron machines that were most known for this. Don't know wha't out there now, tho I remember getting a couple bounces for that reason a couple years back.
there's also an RFC that documents what the supported limits are, and it's a good idea to stay below those...
Yep, its second guessing the MTA, but its a cheap, cost effective, minimal impact guess that has nearly NO punitive effect on mailman itself.
yes and no. By bunching stuff together, you help the MTA optimize, since it's a safe guess that it's going to (at least) sort by domain if it does any kind of connection caching at all.
You could make a good argument that the best way to optimize is to create one mail batch per unique hostname, up to SMTP-MAX-RCPTS, at which point you split it into num_addrs/SMTP-MAX-RCPTS batches for that hostname, and then let the MTA sort if out from there.
I guess that we need a per MTA tuning/configuration document.
Aaaargh. Yes.
Definitely. Since most of the "performance" issues involve the MTA, and the MLM only affects it based on how it stuffs things into the MTA.
Without going and re-reading it, about the only thing I can think to add to it would be turning off domain checking for localhost RCPTs as per our recent comments if that's not there already.
By the way, I suggest that before people *assume* this is an improvement that it be tested, because the domain checking has to be done somewhere -- and if you do it at the MLM->MTA transfer, in most cases it'll still be cached in DNS and save you waiting for the MTA delivery phase. So it's not a matter of avoiding the slowdown, but where you put it -- in the MLM->MTA layer where the person managing the MLM sees it, or in the delivery phase, where it's more or less hidden. But it's still there.
How about Postfix? Anybody know?
Postfix is "on the list" for later this summer for me...
Have we got that data for the other MTAs?
Nope. User written of course, and well, you know.
Sendmail doesn't, sort of. By default, sendmail sorts the queue based on an internal "priority" value based on its view of the "urgency" a piece of mail needs to be delivered with. But you can switch sendmail to use "hostname", where it'll read all of the pending batches, do a domain sort and then try to cache stuff together. Sendmail 8.10 also has a "by filename" option, where it doesn't open the qf file at all, which for diskbound systems can be a big win (I think. I'm still testing out 8.10, and won't move it into production on my big machine until next week; ask me again after that, when I start tuning it) -- but sendmail also does all sorts of connection caching and stuff that confuses the matter, which you can tweak if you want.
Right now, I generally recommend sites doing a lot of mail-list traffic process their queues sorted by hostname, but also run some cron jobs that force queue runs by age, because otherwise, you can have stuff that gets stuck in the queue... This may change under 8.10. I dunno yet.
Of course we could have comparisons of the suitabilities of different MTAs if people enjoy a decent flamefest.
Of course not. Everybody knows that Microsoft Exchange is the one true MTA and all else are but pale imitations.
don't even JOKE about that. As someone who deals with email for a living, the only system that comes *close* to Exchange in the braindead category is Lotus notes. And that's not really close. I have seen so much braindamage out of Exchange servers I wish I could simply reject any mail that ever touched one....
You might as well drive your computers with a squirrel on a wheel.
-- Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com) Apple Mail List Gnome (mailto:chuq@apple.com)
And they sit at the bar and put bread in my jar and say 'Man, what are you doing here?'"
participants (2)
-
Chuq Von Rospach -
J C Lawrence