As an experiment in analysing my system loads I added the following to my exim.conf:

# When this option is set, a delivery process is started whenever a # message is received, directing and routing is performed, and local # deliveries take place. However, if any SMTP deliveries are # required for domains that match queue_smtp_domains, they are not # immediately delivered, but instead the message waits on the queue # for the next queue run. Since routing of the message has taken # place, Exim knows to which remote hosts it must be delivered, and # so when the queue run happens, multiple messages for the same host # are delivered over a single SMTP connection. This option is # checked against the domains supplied in the incoming addresses, # before any widening is done (because that is part of routing). The # -odqs command line option causes all SMTP deliveries to be queued # in this way, and is equivalent to setting queue_smtp_domains to # *'. See also queue_remote_domains, which is subtly different.

queue_smtp_domains = "*"

Note that all of my lists are hand moderated resulting in outbound mail occurring in batches, typically in the range of 5K - 10K MTA spool entries per batch.

The primary effect of adding this config item is that (in general) a compleat moderation batch is enqueued by the MTA before delivery attempts start on any of the component messages. This turns out to have a significant effect on delivery time performance.

Loosely what seems to be happening is that without the above config Exim and Mailman's qrunner end up competing for CPU resources, with system loads climbing into the 30's and 40's with parallel deliveries to the same MX being infrequent to the point of nonexistence. [1] However with the above config Exim has the opportunity to batch all mail targeted to a given MX and parallel deliveries are the default. [2] This shows clearly in the logs with blocks of deliveries to the same MX all together. The apparent result of this batching is that the total delivery time [3] for a moderation batch is now reduced by a factor of almost 5. [3]

Not a bad side effect for a little idle poking about.

-- J C Lawrence
---------(*) Satan, oscillate my metallic sonatas. claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

1. I use 20 queue runners with up to 10 parallel deliveries per MX.

2. I have low rates of duplicate MX use in my subscriber base. AOL + MSN + Hotmail together form less than 3% of my subscriber base. Outside of resource contention this low collision rate is the prime factor in parallel deliveries not happening.

3. Loosely defining total delivery time as the taken to handle all deliveries to all fast MX'es (no errors, warnings, or responses taking more than a second).

4. This depends heavily on whether Mailman's qrunner can get the compleat moderation queue dumped to the MTA before it starts another queue run. Typically for my setup I can get ~4K spool entries queued between MTA queue runs. I *should* be able to get at least an order of magnitude more, and almost two (as determined by trivial test code doing a canned loop). I haven't yet analysed why the qrunner path is so slow and am unlikely to before I deploy v2.1. However as a general observation Mailman v2.0's qrunner seems absurdly slow in iterating across the membership roster and dumping RCPT TO bundles to the MTA.