[Mailman-Users] Increasing the Speed of Email Delivery

[Brad Knowles]

On Dec 23, 2009, at 1:12 PM, Derrick Wooden wrote:

> I'm well aware that SMTP doesn't mean instant.

Regretfully, you are the exception.  Most people don't seem to get this concept -- even people like my wife, and I've been trying to pound this one into her brain for ten years.

> I'm just looking at a solution that can cut 1E6 emails being delivered in 12
> hours down to say 4 hours.  With a fast server, correct MTA optimization and
> proper Mailman setup could this be attainable?

I believe so, yes.  However, optimization would have to be done at all levels, not just the MTA.

If you've got that many recipients, then you're probably seeing a lot of Python pickle contention during initial list delivery, and you'd want to break that down into a number of sub-lists.  If you have a million recipients, then a thousand sublists with a thousand subscribers each would probably be overkill, but a hundred sublists with ten thousand subscribers each probably would not be enough.

Using postfix, you can tune things for maximum parallelization, and eliminate things like unnecessary DNS queries and blacklist checks on outbound (because you've already done them on inbound, or it's an announce-only list and they don't need to be done at all).  You can also tune things so that any messages which don't get delivered right away can get pushed off onto another "slow delivery" server, thus keeping the primary server pumping as fast as possible.  For an announce-only list, you might also want to consider putting the primary mail queues on solid-state disk -- you shouldn't need a particularly large SSD for the primary server if the "slow delivery" server has sufficient storage.

Your primary bottlenecks are going to be disk latency and locking contention.  If you can eliminate or minimize those, everything else should flow as fast as your Internet connection allows.

> The Barack Obama email campaign used Postfix for their MTA and PHP Mailer to
> deliver their newsletters.  

How many people did they deliver to?  How many machines did they have doing delivery?  How often did they send out messages?  How large were those messages?  Did they do DKIM signing on each message?  There's lots of variables here that could affect their numbers relative to what you might be able to achieve.

> I'm using a package install of Mailman along with Exim through cPanel.  I
> now know that this setup is not optimal and needs lots of tweaking.  I'm
> looking into VERP also based on helpful information in the FAQs.

I've done the best I can to encode as much of my knowledge as possible into the FAQs.  Pretty much everything is there, or I point to references where the rest of the information can be found.  The key is knowing what you're looking for and when you've found it.

--
Brad Knowles <bradknowles at shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>