[Mailman-Developers] Huge lists

[Chuq Von Rospach]

At 10:23 PM -0400 5/23/2000, bwarsaw@python.org wrote:

>That would be great.  Right now, I'm not concerned with message sizes,
>but list membership sizes.  If my testing focus were to change, I'd
>ask beforehand.

Message size isn't that significant unless you're doing full VERP. 
And even with VERP, it looks like message size is less important than 
the delays from DNS in resovling slow domains, especially 
international ones.

(this doesn't mean I'd run a 100,000 address list with full VERP on a 
56K line, but assuming you have reasonable network bandwidth for the 
list size, you're more likely to run into DNS delays than anything 
else).

Also, remember that delivery issues are really the MTA's 
responsibility (and problem), not the MLM's -- the important thing 
for Mailman is to pass along the messages in a reasonable matter, and 
let the MTA (sendmail, postfix, etc) worry about optimizing delivery 
-- unless you want to integrate direct delivery into the list 
manager, and it's unclear that really wins you anything.

The important thing is to work on optimizing dropping mail into the 
MTA, and then letting the MTA do its job (and tune it!). In my work, 
the biggest problems is the way MLMs drop mail off for delivery -- 
usually by doing nothing more than a single-threaded drop sorted by a 
reversed address. That puts domain names together (but really ought 
to be a sort by MX instead, it's quite sub-optimal, especially with 
big domain-hosting sites like iname.com), but still limits delivery 
to how fast the MTA will accept addresses, and that's limited by DNS, 
since sendmail resolves addresses as it accepts mail...

So for something like mailman, being able to parallelize delivery 
into the MTA with multiple threads is a lot more important than 
anything else. segregrating slow-resolving domains from faster ones 
would be second, but easier said than done (but you can go a long way 
simply by delivering .com first, .net and .edu second, and then 
everything else...)

I'd guess you can get the first 90% simply by setting up mailman to 
deliver using four threads: .com, .net, .edu/.us/.ca and everything 
else, and allowing people to configure extra threads if the capacity 
allows (and I'd do that by splitting each feed in half), and then 
coming up with guides on how to tune the MTA for fast delivery (I'm 
just starting to figure out sendmail 8.10, but it looks like a nice 
improvement over earlier releases).

-- 
Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com)
Apple Mail List Gnome (mailto:chuq@apple.com)

And they sit at the bar and put bread in my jar
and say 'Man, what are you doing here?'"