On Sat, 4 Nov 2000 10:29:15 -0800 Chuq Von Rospach chuqui@plaidworks.com wrote:

...

...

Once a connetion to the MX is established, bulkmail would then just start delivering messages to it until the bin was emptied. Any i/o blocks in any of the processes will allow async* to switch to a different delivery channel. We may need to do some explicit channel management to make sure some are not starved. Ouch. I really don't like this idea.

Neither do I. This is actually something that I've looked at long and hard in my non-mailman server work. After a fair amount of work and research, I finally came to the conclusion that you are MUCH better off letting the MTA do the MTA's work, and letting the MLM do the MLM work, and once you make the decision that the MLM has to *also* become an MTA, you're doing down a road you don't want to travel.

Agreed. There's also the simple aspect of the fact that optimising for the very large case (which needs the MLM-sepcific MTA supports ala ListServ) is near on pessimal for the small and medium cases. Spiffy domain based routing and near-target explosion just don't make a whole lot of sense for lists with a few hundred/thousand members. Asides from which, the optimisations for mail delivery to a massive number of targets and mass mail delivery to a comparitively constrained target set can be rather different.

That's ugly territorry. We don't need to go there yet.

Instead of reinventing the MTA wheel, I think we're much better off coming up with an MTA -> MLM interface that's very flexible and highly configurable (most especially in how to deliver and how much to parallelize the infeed to the MLM), and then focus on how to tune the MTA and MLM through documentation.

Agreed. There's not a whole lot of tuning you can do on SMTP hand-offs to the MTA (which is necessary as a feature to support non-localhost MTAs which is in turn needed by many sites), but some MTAs have divers command line options that can be useful in an MLM delivery world (peeking quickly at the Exim docs).

Splitting the inbound and outbound queue would be my first thing here, and probably split bounces into a third queue. That's a pretty quick, easy optimization that makes sure the end user sees fast response without being bogged down by deliveries, and that's a huge perception issue. Then focus on parallelizing the delivery from mailman into the MTA, and make that configurable so each admin can tune it to their system and needs.

<nod>

Without spending any time collecting metrics, it does look like the low hanging fruit is in handling the MLM->MTA handoff. Given that the distribution of posts across lists is often uneven (and unpredictable), and that the optimal parallelism (N) is fairly fixed for a given machine, any pattern which supported keeping N pipes stuffed throwing things at the MTA (with minimal locking) would seem fair.

As for splitting the queues, yep, hadn't thought about that aspect, but that's an obvious contention area for an active list.

...

As discussed previously amongst Chuq, Nigel and I, the needs of large list server systems are rather different from the normal home hobbyest requirements, but are not compleatly alien. However, the needs of very large list installations (cf ListServ, Egroups, or SourceForge) are rather different yet again.

This is a basic reality -- things don't scale. Or worse, they scale for a while, and then you need to switch paradigms. I found that one out the hard way. If someone wants a rhetoric on how to scale mail list servers infinitely, I'd be happy to explain how, since I've had to develop an architecture to do so. the nice thing is, it can be done without exceptional engineering hassles -- but it's not just adding another daemon or a faster CPU (although those are solutions for parts of it, just not ALKWAYs the solutions)

We should really have lunch some time. I'm currently working down off San Tomas and Saratoga (startup, oddly enough next door to BeOpen). I'm wrapping up this contract this month and looking for next one, so time is a little tight here, but I'd love to get together for a chat. You over at the main Apple campus?

...

I'm not convinced of the value in beating on Mailman to support the (comparitively rare if high profile) very large installations when the current (much larger and more common Mailman-wise) mid-size realm still needs attention. Certainly, such changes should not detract from Mailman's current level of suitability for smaller installations.

I think we can build a Mailman that does this, at least for, oh, 95% of the universe out there, and the other 5% are going to have custom solutions anyway (or should!). What we don't want to do is screw up Mailman for the "typical" user to make it work for the big site; but we also don't want Mailman to get a reputation as a "small server only" system, because it'll cause people to reject it in implementations. Fortunately, I don't think you need to do that. It just needs some tweaking.

<nod>

We must stop agreeing like this. Its getting embarassing.

And for sites this *doesn't* solve, it's either because they're doing the 5 pounds in a 1 pound bag thing, or they probably need to start hiring people like us to custom build someething.

Ungghhhh.

-- J C Lawrence Home: claw@kanga.nu ---------(*) Other: coder@kanga.nu http://www.kanga.nu/~claw/ Keys etc: finger claw@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--