SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES?

-----Original Message----- From: Mark Sapiro Sent: Saturday, June 25, 2011 8:39 PM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration..

Khalil Abbas wrote:

we are a local TV station & newspaper having a mailing list with 2+ Million Subscribers.. what is the best server configuration (CPU, RAM) that can handle this list? taking in consideration that only 1 daily email is sent to the subscribers..

Consider that likely much more than half the impact is going to be from the MTA rather than Mailman. Perhaps you should be asking on a list specific to your MTA.

I would suggest at a minimum that you could reduce your Mailman impact by breaking up your list into perhaps 240 roughly equal sub-lists and sending the post to one list at a time at 6 minute intervals. Then if your system can complete delivery of the post from one sub-list to its 8500+- members in under 6 minutes, you should be OK.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Khalil Abbas wrote:

SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES?

WHY ARE YOU SHOUTING?

I have no real information as to why Hotmail and Yahoo were not accepting your mail, but apparently at one point I thought that reducing the number of Hotmail and/or Yahoo envelope recipients in a single message transaction might help and apparently, it did. Presumably, nothing at Hotmail and Yahoo has changed, so I wouldn't advise changing SMTP_MAX_RCPTS.

What I would advise is before changing other things including hardware, that you try the Postfix changes recommended to you at http://mail.python.org/pipermail/mailman-users/2011-June/071671.html which you apparently never tried or at least never reported trying, and if that doesn't solve the problem completely, that you try the patch in the attached throttle.patch.txt file. The settings in this patch will limit Mailman to send to 2000 recipients per minute which will spread your 2+ million recipients over about 17 hours and will be as effective as making many small lists.

Note that contrary to your assertion at http://mail.python.org/pipermail/mailman-users/2011-June/071720.html that the patch is "not stable and not tested before", I never said it was not tested. The FAQ says it "has been only minimally tested and is not guaranteed to work". What that means is it hasn't seen significant use and it may not solve your problem, but it is almost certainly more stable than rebooting the server every hour.

And if you do try it and it does work, we can remove the caveats, and if it doesn't solve th problem, at least we'll know that.

Free, open source software is a community effort, not a one-way street.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

lol I'm not shouting, just too lazy to hit caps :)

well I bought this server:

RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore

I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes ..

there was about 500 yahoo addresses and yahoo deferred a few messages.. luckily Hotmail accepted all 5500 addresses..

is that good?

do you think getting 2 or 3 or even 4 servers like this one and splitting the 2 million users over them is better?

about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to install the patch :)

I have mailman 2.1.9 by the way ..

Thanks ..

-----Original Message----- From: Mark Sapiro Sent: Saturday, June 25, 2011 10:40 PM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration..

Khalil Abbas wrote:

SMTP_MAX_RCPTS =5 WOULD BE OK? OR SHOULD I REDUCE OR INCREASE IT FOR HOTMAIL AND YAHOO TO ACCEPT OUR MESSAGES?

WHY ARE YOU SHOUTING?

I have no real information as to why Hotmail and Yahoo were not accepting your mail, but apparently at one point I thought that reducing the number of Hotmail and/or Yahoo envelope recipients in a single message transaction might help and apparently, it did. Presumably, nothing at Hotmail and Yahoo has changed, so I wouldn't advise changing SMTP_MAX_RCPTS.

What I would advise is before changing other things including hardware, that you try the Postfix changes recommended to you at http://mail.python.org/pipermail/mailman-users/2011-June/071671.html which you apparently never tried or at least never reported trying, and if that doesn't solve the problem completely, that you try the patch in the attached throttle.patch.txt file. The settings in this patch will limit Mailman to send to 2000 recipients per minute which will spread your 2+ million recipients over about 17 hours and will be as effective as making many small lists.

Note that contrary to your assertion at http://mail.python.org/pipermail/mailman-users/2011-June/071720.html that the patch is "not stable and not tested before", I never said it was not tested. The FAQ says it "has been only minimally tested and is not guaranteed to work". What that means is it hasn't seen significant use and it may not solve your problem, but it is almost certainly more stable than rebooting the server every hour.

And if you do try it and it does work, we can remove the caveats, and if it doesn't solve th problem, at least we'll know that.

Free, open source software is a community effort, not a one-way street.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Stefan Foerster wrote:

Go with SMTP_MAX_RCPT=1000 in Mailman but limit the number of Postfix processes, start with 20 (postconf -e default_process_limit=20 && postfix reload), see how that works out for you and increase the default_process_limit if it works well until performance gets worse.

Part of the OP's original issue is that with the default SMTP_MAX_RCPTS = 500, messages delivered from Mailman to Postfix had hundreds of Hotmail (and/or Yahoo) envelope recipients. Postfix would apparently then deliver the message to Hotmail, again in one transaction with all the Hotmail envelope recipients, and Hotmail would not accept an envelope with that many recipients (and similarly for Yahoo).

Probably, SMTP_MAX_RCPTS could be larger than 5 and still avoid the problem, but it probably can't be more than 20 or so.

Of course, the real solution is for the OP to negotiate with Microsoft and Yahoo and maybe others to get his server whitelisted if he can.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

this sucks! I've created a list of 100,000 subscribers .. SMTP_MAX_RCPTS=5, throttle worked perfectly, 800 emails per minute.. the messages should be delivered in a 2 hours time .. now it's 4 hours and 47,000 messages are stuck because hotmail n yahoo are deferring ..

I read somewhere that the biggest list that's using mailman is 500,000 in size.. is it really true or only 50,000 messages are going thru n the rest is delayed or even bounced back..

I don't know what to do ... I did everything in the book.. our viewers had subscribed to our lists with double opt-in .. mailman is properly configured.. I don't know what yahoo and hotmail really want! I'm so desperate and disappointed..

**sigh**

-----Original Message----- From: Mark Sapiro Sent: Sunday, June 26, 2011 5:59 AM To: Khalil Abbas ; mailman-users@python.org Subject: Re: [Mailman-Users] Server Configuration..

Khalil Abbas wrote:

lol I'm not shouting, just too lazy to hit caps :)

Regardless of why it happened, it comes across to others as shouting.

well I bought this server:

RAM 1x 4 GB DDR2-RAM ECC HDD 2x 500 GB SATA II-HDD 7.200 rpm Barebone 1x Fujitsu PRIMERGY RX100 S5 CPU 1x Intel Xeon X3320 Quadcore

I did set the SMTP_MAX_RCPT to 5 .. then I created a list of 10,000 members, n it delivered them in like 5-6 minutes ..

there was about 500 yahoo addresses and yahoo deferred a few messages.. luckily Hotmail accepted all 5500 addresses..

is that good?

It's not terrific. At that rate, assuming everything had completely left your MTA in that time, it would take between 17 and 20 hours to deliver to 2 million recipients. Is that acceptable?

Also, it is not clear that just because Hotmail accepted all 5500 addresses out of 10,000, that they will accept the next 55,000 out of the next 100,000 (if there are that many). Also, once Yahoo starts deferring, their deferral rate will increase if you keep sending more, and this will add to the load on your MTA.

do you think getting 2 or 3 or even 4 servers like this one and splitting the 2 million users over them is better?

I can't say. Perhaps the bottleneck is the bandwidth of your internet connection. If so, increasinfg the ability to push more data faster won't help at all.

about the throttle patch, ok I confess, I AM IGNORANT! I have no idea how to install the patch :)

I have mailman 2.1.9 by the way ..

The throttle.patch.txt file attached to this message contains a 2.1.9 version of the patch. The only difference is the line number of the last portion.

To install it, save the file and cd to your mailman installation and give the command

patch Mailman/Handlers/SMTPDirect.py < /path/to/throttle.patch.txt

and then restart Mailman.

If anything goes wrong, you can remove the patch with the command

patch -R Mailman/Handlers/SMTPDirect.py < /path/to/throttle.patch.txt

followed by another restart of Mailman.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Khalil Abbas wrote:

this sucks! I've created a list of 100,000 subscribers .. SMTP_MAX_RCPTS=5, throttle worked perfectly, 800 emails per minute.. the messages should be delivered in a 2 hours time .. now it's 4 hours and 47,000 messages are stuck because hotmail n yahoo are deferring ..

We cannot help you directly. Hotmail and Yahoo think you are a spammer. You have to negotiate with them directly and convince them that you aren't and that their customers want your mail and get them to whitelist your server.

I read somewhere that the biggest list that's using mailman is 500,000 in size.. is it really true or only 50,000 messages are going thru n the rest is delayed or even bounced back..

It is probably true, but it is also probably true that there aren't hundreds of thousands of Hotmail addresses on that list.

I don't know what to do ... I did everything in the book.. our viewers had subscribed to our lists with double opt-in .. mailman is properly configured.. I don't know what yahoo and hotmail really want! I'm so desperate and disappointed..

How are they supposed to know your list is double opt-in if you don't talk to them? See the FAQ at http://wiki.list.org/x/4oA9. You are going to have to contact them and convince them to whitelist you if you can.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 6/26/2011 11:41 PM, Ralf Hildebrandt wrote:

With Yahoo!, this is relatively easy: http://feedbackloop.yahoo.net/index.php?page=%2Fmanage.php

With Hotmail you need to go to https://postmaster.live.com/

Thanks Ralf. I added the Yahoo link to the FAQ at http://wiki.list.org/x/4oA9. I had added a Hotmail link yesterday, and I added a bit more info to it too.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan