[Mailman-Developers] Re: [Mailman-Users] Problem with qrunner and too much incoming mail

[J C Lawrence]

On Sat, 4 Nov 2000 10:29:15 -0800 
Chuq Von Rospach <chuqui@plaidworks.com> wrote:

>> > Once a connetion to the MX is established, bulkmail would then
>>> just start delivering messages to it until the bin was emptied.
>>> Any i/o blocks in any of the processes will allow async* to
>>> switch to a different delivery channel.  We may need to do some
>>> explicit channel management to make sure some are not starved.
>>  Ouch.  I really don't like this idea.

> Neither do I. This is actually something that I've looked at long
> and hard in my non-mailman server work. After a fair amount of
> work and research, I finally came to the conclusion that you are
> MUCH better off letting the MTA do the MTA's work, and letting the
> MLM do the MLM work, and once you make the decision that the MLM
> has to *also* become an MTA, you're doing down a road you don't
> want to travel.

Agreed.  There's also the simple aspect of the fact that optimising
for the very large case (which needs the MLM-sepcific MTA supports
ala ListServ) is near on pessimal for the small and medium cases.
Spiffy domain based routing and near-target explosion just don't
make a whole lot of sense for lists with a few hundred/thousand
members.  Asides from which, the optimisations for mail delivery to
a massive number of targets and mass mail delivery to a
comparitively constrained target set can be rather different.  

That's ugly territorry. We don't need to go there yet.

> Instead of reinventing the MTA wheel, I think we're much better
> off coming up with an MTA -> MLM interface that's very flexible
> and highly configurable (most especially in how to deliver and how
> much to parallelize the infeed to the MLM), and then focus on how
> to tune the MTA and MLM through documentation.

Agreed.  There's not a whole lot of tuning you can do on SMTP
hand-offs to the MTA (which is necessary as a feature to support
non-localhost MTAs which is in turn needed by many sites), but some
MTAs have divers command line options that can be useful in an MLM
delivery world (peeking quickly at the Exim docs).

> Splitting the inbound and outbound queue would be my first thing
> here, and probably split bounces into a third queue. That's a
> pretty quick, easy optimization that makes sure the end user sees
> fast response without being bogged down by deliveries, and that's
> a huge perception issue. Then focus on parallelizing the delivery
> from mailman into the MTA, and make that configurable so each
> admin can tune it to their system and needs.

<nod>

Without spending any time collecting metrics, it does look like the
low hanging fruit is in handling the MLM->MTA handoff.  Given that
the distribution of posts across lists is often uneven (and
unpredictable), and that the optimal parallelism (N) is fairly fixed
for a given machine, any pattern which supported keeping N pipes
stuffed throwing things at the MTA (with minimal locking) would seem
fair.

As for splitting the queues, yep, hadn't thought about that aspect,
but that's an obvious contention area for an active list.

>> As discussed previously amongst Chuq, Nigel and I, the needs of
>> large list server systems are rather different from the normal
>> home hobbyest requirements, but are not compleatly alien.
>> However, the needs of very large list installations (cf ListServ,
>> Egroups, or SourceForge) are rather different yet again.

> This is a basic reality -- things don't scale. Or worse, they
> scale for a while, and then you need to switch paradigms. I found
> that one out the hard way. If someone wants a rhetoric on how to
> scale mail list servers infinitely, I'd be happy to explain how,
> since I've had to develop an architecture to do so. the nice thing
> is, it can be done without exceptional engineering hassles -- but
> it's not just adding another daemon or a faster CPU (although
> those are solutions for parts of it, just not ALKWAYs the
> solutions)

We should really have lunch some time.  I'm currently working down
off San Tomas and Saratoga (startup, oddly enough next door to
BeOpen).  I'm wrapping up this contract this month and looking for
next one, so time is a little tight here, but I'd love to get
together for a chat.  You over at the main Apple campus?

>> I'm not convinced of the value in beating on Mailman to support
>> the (comparitively rare if high profile) very large installations
>> when the current (much larger and more common Mailman-wise)
>> mid-size realm still needs attention.  Certainly, such changes
>> should not detract from Mailman's current level of suitability
>> for smaller installations.

> I think we can build a Mailman that does this, at least for, oh,
> 95% of the universe out there, and the other 5% are going to have
> custom solutions anyway (or should!). What we don't want to do is
> screw up Mailman for the "typical" user to make it work for the
> big site; but we also don't want Mailman to get a reputation as a
> "small server only" system, because it'll cause people to reject
> it in implementations. Fortunately, I don't think you need to do
> that. It just needs some tweaking.

<nod>

We must stop agreeing like this.  Its getting embarassing.

> And for sites this *doesn't* solve, it's either because they're
> doing the 5 pounds in a 1 pound bag thing, or they probably need
> to start hiring people like us to custom build someething.

Ungghhhh.

-- 
J C Lawrence                                 Home: claw@kanga.nu
---------(*)                               Other: coder@kanga.nu
http://www.kanga.nu/~claw/        Keys etc: finger claw@kanga.nu
--=| A man is as sane as he is dangerous to his environment |=--