[Mailman-Developers] Huge lists

[J C Lawrence]

On Thu, 25 May 2000 00:29:48 -0700 
Chuq Von Rospach <chuqui@plaidworks.com> wrote:

> At 12:05 AM -0700 5/25/2000, J C Lawrence wrote:

>> Its tough to image a situation where my time and effort in
>> replacing them (as a solo effort) would actually be worth it as
>> versus throwing hardware at the problem or chatting up Wietse &
>> co.

> throwing hardware at a problem isn't always possible. 

True, its an argument for mid-range problems: problems that aren't
so large that you're only effective recourse is optimisation.

> but the place where rolling your own internal MTA starts becoming
> useful is when the list is big enough that the disk I/O involving
> the MTA starts becoming the significant limiter. 

Arguably disk IO is the only limiting factor in an MTA which you
have total control over.  Everything else, network latency, DNS,
remote MTAs etc etc etc are out of the hands of the MTA author.

This is one of the reasons that CP uses solid state disks for their
mail spools.  Even with QMail's quite optimised disk IO, it still
hurts.

> VERP exacerbates the problem, since # of batches sent to the MTA
> equals the # of addresses, which explodes the number of control
> files, which... 

Exactly, which is why i proposed in the asynch model that the
handler process only feed mesages to the MTA at no more than a set
rate.  That way the local SysAdm can set the rate to something that
his MTA is likely to be able to more or less keep up with (or at
least not fall too far behind on).  

Tough balancing point there.  Too fast and you drown the MTA.  Too
slow and you starve and delivery time is needlessly extended.
Getting it right all the time is impossible due to remote delivery
problems being unpredictable and the absence of any feedback
mechanisms between a fully abstracted MLM and the MTA.

This is one of the ways in which EZMLM has it easy if what I've been
told is correct.  I've been told (haven't checked) that it kinda
fakes VERP, not exploding the spool, but build the messages with the
VERP contents only upon MX connection.  Prior to that its just a
template with a long attached address list.  That makes it really
cheap.  But then again, that's due to the fact that EXMLM is
explicitly tied to QMail...

> So at some point, it makes sense to deliver direct to recipient
> rather than build batches into the MTA, and completely avoid the
> disk I/O and deliver right out of the database to the receiving
> SMTP client. You could strongly parallelize the delivery setup
> because you'd do away with all of the MTA overhead, and do all
> sorts of fun things, like prioritize your delivery sorting and the
> like.

Yeah, all sorts of fun things, like, in essence, writing the entire
delivery and queue handling side of a normal MTA, with the single
extention of doing last-second runtime generation of VERP messages
qt the instant you send them down the wire (failed delivery?  trash
the temp VERP message and leave that address as "undelivered").

Yep, you'll need that for very large mailing lists (eg >1M
subscribers and >1 post per hour).  But, I really don't think that's
Mailman's audience.  That's LServe's audience.  There are damned few
lists out there that big...

> Which, if you're trying to deliver 5,000,000 emails a day and do
> so within a time-sensitive time period gets important -- and for
> the other 99.5% of the universe, just doesn't matter that much
> (snork).

Bingo.

>> True.  Were Mailman asynchronous, a pattern as below would seem
>> useful:
>> 
>> There is never more a single "queued message handler" process
>> (maybe multi-threaded, or not).  That process guarantees not to
>> feed messages to the MTA any faster than XXX messages per
>> second/minute, and to stop such feeding were system load to rise
>> above ZZZ.  The single instance rule prevents multiple handler
>> processes for multiple mailing lists maxxing out the MTA as they
>> all dump simultaneously.

Above quoted only to provide reference to the earlier text on queue
handling.

> Oh, queuing theory is such fun. I got into computers to AVOID
> math...

Heck, I got out of math because I was too damned lazy...

>> The problem of multiple list servers (boxes) dumping
>> simultaneously to a remote MTA is properly, I believe, outside of
>> Mailman's purview.
>> 
>> I don't see a value in trying to monitor MTA queue size.  Too MTA
>> specific.

> See the disk I/O issues above. In a perfect world, the MTA would
> self-throttle itself to avoid overload conditions. In practice,
> you have to be careful to both tune the MTA to maximize output,
> and the MLM to avoid blowing it out. If you have a burst that
> stuffs 2500 batches into a sendmail queue all at once, then
> sendmail has that big directory problem i a big way, and your
> system goes to hell.

While true, that's a sticky wicket.  Mailman is MTA agnostic (a Good 
Thing).  Yup, you can build all sorts of intelligence and queue
handling techniques in, but the more you do there, the less and less 
clean (or possible) your abstractions are going to be, and the more
you are going to tie yourself into a specific configuration.  

We could, without too too much effort, whack Mailman into being,
say, Postfix specific and actively conspiring at runtime with
Postfix for optimal queue delivery yada yada whoopdedoo yabba yabba
and we'd get something that could potentially easily handle a
million subscriber list with a post an hour.

But we don't want to do that.  EZMLM has already done that.  Last I
checked there was a nascent project to a Postfix-specific MLM ala
EZMLM (never noticed if it got off the ground).  

> I wonder how much of this could be driven out of something like
> Midgard? But loading your entire archives into a database gives
> you the ability to do all sorts of interesting linking and
> searching and stuff, and "all" you'd need is some email->XML
> converter, and then...

I'm already kinda busy on another side of this:

  http://www.kanga.nu/archives/Meta-L/2000Q2/msg00242.html
  http://www.kanga.nu/archives/Meta-L/2000Q2/msg00246.html

> Oh, man. We need to at least pretend to be on topic for this list,
> but I need a white board and a pen... (scribbly scribble...)

Gods, I need three of me.

-- 
J C Lawrence                              Internet: claw@kanga.nu
----------(*)                            Internet: coder@kanga.nu
...Honorary Member of Clan McFud -- Teamer's Avenging Monolith...