[Mailman-Developers] Some notes on my stuff...

[Graham TerMarsch]

On Monday 23 July 2001 08:41, Bill Bradford wrote:
> On Sun, Jul 22, 2001 at 10:15:15PM -0700, Chuq Von Rospach wrote:
> > My testing showed the major processing delay was the speed of uptake
> > by sendmail, primarily due to DNS delays. And given that you can't
> > turn off DNS lookups without turning off a lot of the anti-spam stuff,
> > you're kinda stuck.
>
> I went from sendmail to postfix, and saw at least a 10x speed
> improvement (from 12-hours-max from sending an email to a list to
> receiving it, to 12 *minutes*.)  Turns out Sendmail was choking on the
> "backlog" queue of deferred/undeliverable email.  Postfix basically says
> "okay, I"ll try this later, and i'll deliver the email now that CAN be
> delivered".

>From some of the "announcement only" lists that I manage for my customers, 
we found similar things, but managed to do it by firing up different 
copies of Sendmail periodically to run the mail queue.  We too had similar 
problems with Sendmail stalling when it'd hit some domain that didn't 
resolve properly or that took forever to connect to, and it slowed the 
whole process right down.

After doing some tweaking and twiddling, we've now got Sendmail configured 
with several different types of queue runners that we fire up via cron:

- One that runs with really _short_ timeouts, to go deep into the mail 
queue and send stuff out as quick as possible, without bothering to wait 
for long DNS resolves or slow net connections.

- One that runs with really _long_ timeouts, to try to process those msgs 
going to hosts that just happen to be on the end of really slow links on 
the net.

- And a whole bunch in the middle that we fire up to process mail based on 
destination domain.  By doing a "mailq" and grabbing/counting the 
destination domains, we can figure out where most of the mail is going, 
and then use "sendmail -qR" to fire up one (or more) queue runners to 
deliver mail out to those domains.

It did take us a bit of twiddling to get the queue runners to fire up and 
send things out at full tilt without overloading the machine, but after we 
got those wrinkles all worked out, we found that Sendmail worked extremely 
well (and _MUCH_ better than it did with an "out of the box" config).

Also found that two Sendmail config options that made a _TON_ of 
difference, were "HostStatusDirectory" and "SingleThreadDelivery".  The 
first one lets you share your host up/down status amongst all of the 
running Sendmail procs, while the other keeps you from overloading any one 
particular destination server by only allowing a single connection to it 
from your end.

------------------------------------------------------------

And lastly, I think I've got to agree wholeheartedly with Chuq on his 
notes about spindle usage and IO bandwidth being one of (if not _the_) big 
limiting factors in delivering mail.  This same machine I've discussed 
above saw a fourfold improvement in mail throughput by pulling the mail 
spool off onto an underused spindle.

I know Sendmail is generally chastized as being slow and doggish as far as 
SMTP servers go, but from having twiddled with it enough lately, I'm 
beginning to wonder if thats just because the default config that it 
builds with is sluggish (as I've been able to get ~8-10x throughput from 
the same server just from reshuffling and retuning the config and the 
queue runners).

-- 
Graham TerMarsch