Hi,
I think there's a small design flaw with the once-in-a-wile VERPing scheme. My biggest list is 180k subscribers, and I've set up Mailman to VERP once every 10th message. Well, it happened today that the big list was hit by its VERP time, and it's a bit awful - it looks like the list has taken control of Mailman, and no other mail can pass through it.
I think it might be wiser to say "every 10th recipient" is VERPed at each sending, than what we have now. Of course, it needs rotating things, or randomizing them, so it's not a two-lines of code I guess.
Meanwhile I'll just have to disable VERP for this list (or for the whole server).
-- Fil
The problem is you cannot disable the VERP bounce probe (ping) even if you disable VERP in the mm_cfg.py
MANA TANGATA
-----Original Message----- From: Fil [mailto:fil@rezo.net] Sent: Monday, July 12, 2004 2:15 PM To: Mailman Developers Subject: [Mailman-Developers] VERPing: ouch!
Hi,
I think there's a small design flaw with the once-in-a-wile VERPing scheme. My biggest list is 180k subscribers, and I've set up Mailman to VERP once every 10th message. Well, it happened today that the big list was hit by its VERP time, and it's a bit awful - it looks like the list has taken control of Mailman, and no other mail can pass through it.
I think it might be wiser to say "every 10th recipient" is VERPed at each sending, than what we have now. Of course, it needs rotating things, or randomizing them, so it's not a two-lines of code I guess.
Meanwhile I'll just have to disable VERP for this list (or for the whole server).
-- Fil
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/somu chfun%40atlantismail.com
At 3:24 PM -0700 2004-07-12, Somuchfun wrote:
The problem is you cannot disable the VERP bounce probe (ping) even if you disable VERP in the mm_cfg.py
This is assuming you're using Mailman 2.1.5. See also <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq05.012.htp>.
Moreover, this is just the bounce management system, which should be many, many orders of magnitude smaller than the largest list on the system.
-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
At 11:14 PM +0200 2004-07-12, Fil wrote:
I think there's a small design flaw with the once-in-a-wile VERPing scheme. My biggest list is 180k subscribers, and I've set up Mailman to VERP once every 10th message. Well, it happened today that the big list was hit by its VERP time, and it's a bit awful - it looks like the list has taken control of Mailman, and no other mail can pass through it.
Well, a 180k list is pretty big by Mailman standards. That's the largest list in current operations that I know of. There have been larger lists in the past, but I haven't heard of any this size or larger since.
A big list is going to be painful to send out, if you don't have the hardware sized and configured correctly for this. Search the FAQ wizard for the word "performance" and read every entry returned, if you want to get some idea of what this entails.
I think it might be wiser to say "every 10th recipient" is VERPed at each sending, than what we have now. Of course, it needs rotating things, or randomizing them, so it's not a two-lines of code I guess.
As you point out, this is a non-trivial change.Meanwhile I'll just have to disable VERP for this list (or for the whole server).
I imagine that disabling VERP for this one list should probably be enough. If it wasn't, you would have been coming to us already asking about how you can tune your performance.
-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
I think there's a small design flaw with the once-in-a-wile VERPing scheme. My biggest list is 180k subscribers, and I've set up Mailman to VERP once every 10th message. Well, it happened today that the big list was hit by its VERP time, and it's a bit awful - it looks like the list has taken control of Mailman, and no other mail can pass through it.
In fact I bragged: it's only 152991 subs :)
Here's the smtp log line: Jul 13 02:05:22 2004 (435) <20040712150834.DCA153680BD@alan.rezo.net> smtp for 152991 recips, completed in 31782.241 seconds
At 02:05 suddenly all the other messages that were waiting to be sent were sent, and the server went into unstable mode (my graphics logs did panic for a while), then everything reverted to normal.
In any case, 2 seconds per VERP message for such a big list is too costly, so I'll just disable VERPing the normal messages for now - with the price of not being able to clean the list as efficiently as I could with the occasional VERP.
I hope this feedback can be useful, if you have scalability in mind for MM3.
-- Fil
At 11:08 AM +0200 2004-07-13, Fil wrote:
Here's the smtp log line: Jul 13 02:05:22 2004 (435) <20040712150834.DCA153680BD@alan.rezo.net> smtp for 152991 recips, completed in 31782.241 seconds
How does this compare to a normal, non-VERPed delivery for this list?
I ask because Chuq Von Rospach has done some calculations on what should theoretically happen to your performance if you enable VERP, but I don't know of anyone who has actually timed the performance difference on large lists.
In any case, 2 seconds per VERP message for such a big list is too costly, so I'll just disable VERPing the normal messages for now - with the price of not being able to clean the list as efficiently as I could with the occasional VERP.
I hope this feedback can be useful, if you have scalability in mind for MM3.
This is really more of an MTA limitation, although there might be some things we can do to try to work around it with Mailman. For example, it might be faster/lower overall load on the server if we had the MTA do the VERPing for us -- we're pretty sure that's supported by some MTAs (e.g., at least some versions of Exim), and we know it's faster for at least some of them (e.g., Exim).
-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
for 152991 recips, completed in 31782.241 secondsHow does this compare to a normal, non-VERPed delivery for this list?
grep -E "1..... recips" logs/smtp
May 27 16:43:46 2004 (440) <20040527135407.74A5F368143@alan.rezo.net> smtp for 151942 recips, completed in 1231.438 seconds Jun 11 19:05:45 2004 (440) <20040611163245.ED9CB3680AE@alan.rezo.net> smtp for 152333 recips, completed in 649.634 seconds Jun 30 15:39:26 2004 (435) <20040630132741.F1A0836811C@alan.rezo.net> smtp for 152717 recips, completed in 428.891 seconds Jul 13 02:05:22 2004 (435) <20040712150834.DCA153680BD@alan.rezo.net> smtp for 152991 recips, completed in 31782.241 seconds
I ask because Chuq Von Rospach has done some calculations on what should theoretically happen to your performance if you enable VERP, but I don't know of anyone who has actually timed the performance difference on large lists.
Usually the sending (mailman to postfix to 90% of users) takes a bit more than two hours ; yesterday it took about 6 hours. But more importantly, the Mailman -> postfix thing took 5 hours instead of ~ 15 minutes.
This is really more of an MTA limitation, although there might be some things we can do to try to work around it with Mailman. For example, it might be faster/lower overall load on the server if we had the MTA do the VERPing for us -- we're pretty sure that's supported by some MTAs (e.g., at least some versions of Exim), and we know it's faster for at least some of them (e.g., Exim).
Probably, yes. I don't konw if postfix can do it "on demand", though there is http://www.postfix.org/VERP_README.html
-- Fil
At 2:28 PM +0200 2004-07-13, Fil wrote:
for 152991 recips, completed in 31782.241 secondsHow does this compare to a normal, non-VERPed delivery for this list?
grep -E "1..... recips" logs/smtp
May 27 16:43:46 2004 (440) <20040527135407.74A5F368143@alan.rezo.net> smtp for 151942 recips, completed in 1231.438 seconds
1231.438/151942 = 0.0081046 seconds per recipient (average)Jun 30 15:39:26 2004 (435) <20040630132741.F1A0836811C@alan.rezo.net> smtp for 152717 recips, completed in 428.891 seconds
428.891/152717 = 0.0028084 seconds per recipient (average)Jul 13 02:05:22 2004 (435) <20040712150834.DCA153680BD@alan.rezo.net> smtp for 152991 recips, completed in 31782.241 seconds
31782.241/152991 = 0.2077392 seconds per recipient (average)
So, on average, you're taking somewhere close to 25-75 times as long, if you enable VERP for the entire list. Ouch!
I ask because Chuq Von Rospach has done some calculations on what should theoretically happen to your performance if you enable VERP, but I don't know of anyone who has actually timed the performance difference on large lists.
Usually the sending (mailman to postfix to 90% of users) takes a bit more than two hours ; yesterday it took about 6 hours. But more importantly, the Mailman -> postfix thing took 5 hours instead of ~ 15 minutes.
I will definitely update the VERP performance entry in the FAQ to reference your experience.
-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
At 3:26 PM +0200 2004-07-13, Brad Knowles wrote:
Usually the sending (mailman to postfix to 90% of users) takes a bit more than two hours ; yesterday it took about 6 hours. But more importantly, the Mailman -> postfix thing took 5 hours instead of ~ 15 minutes.
I will definitely update the VERP performance entry in the FAQ to reference your experience.
Done. See <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.012.htp>. Please let me know if you think that any other information needs to be added, or if there are any corrections you feel need to be made.
-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
Fil <fil@rezo.net> writes:
Probably, yes. I don't konw if postfix can do it "on demand", though there is http://www.postfix.org/VERP_README.html
It does if you pass it the right options.
Rather than having mailman do the VERPing, I let postfix handle it. Performance is greatly improved, because at the least you can then have multiple recipients per queue file.
There's several things mailman should be letting the MTA handle, as long as it can. VERP and requesting no warning messages, for example.
Darrell
At 9:54 AM -0700 2004-07-13, Darrell Fuhriman wrote:
Rather than having mailman do the VERPing, I let postfix handle it. Performance is greatly improved, because at the least you can then have multiple recipients per queue file.
Can you tell us what modifications you've made to mailman to make it do this correctly with postfix?
There's several things mailman should be letting the MTA handle, as long as it can. VERP and requesting no warning messages, for example.
Agreed. Got patches?-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
For example, it might be faster/lower overall load on the server if we had the MTA do the VERPing for us -- we're pretty sure that's supported by some MTAs (e.g., at least some versions of Exim), and we know it's faster for at least some of them (e.g., Exim).
Sorry, for postfix indeed it might be much easier to do it this way: http://www.postfix.org/VERP_README.html#smtp
the only change between non-verp and verp call to the SMTP server is to replace MAIL FROM:<sender@domain> by MAIL FROM:<sender@domain> XVERP=+=
-- Fil
At 2:32 PM +0200 2004-07-13, Fil wrote:
Sorry, for postfix indeed it might be much easier to do it this way: http://www.postfix.org/VERP_README.html#smtp
the only change between non-verp and verp call to the SMTP server is to replace MAIL FROM:<sender@domain> by MAIL FROM:<sender@domain> XVERP=+=
I don't know what this would do for performance with postfix (relative to having Mailman do the VERPing), but it would certainly be interesting to find out.
Can you modify your code so as to generate this modified string, and see what happens?
-- Brad Knowles, <brad.knowles@skynet.be>
"Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755SAGE member since 1995. See <http://www.sage.org/> for more info.
Sorry, for postfix indeed it might be much easier to do it this way: http://www.postfix.org/VERP_README.html#smtp
the only change between non-verp and verp call to the SMTP server is to replace MAIL FROM:<sender@domain> by MAIL FROM:<sender@domain> XVERP=+=
I don't know what this would do for performance with postfix (relative to having Mailman do the VERPing), but it would certainly be interesting to find out.
Can you modify your code so as to generate this modified string, and see what happens?
It's not so easy for me: it's a production server, and I'm bad at python :)
Moreover, and more importantly, what needs to be done is first to test the presence of XVERP in the SMTPd server we connect to. Postfix does not enable it by default. Then check if the message wants to be VERP (but not decorated). Then only ask XVERP from the MTA. Too ambitious for me...
-- Fil
On Wed, 2004-07-14 at 01:28, Brad Knowles wrote:
At 2:32 PM +0200 2004-07-13, Fil wrote:
the only change between non-verp and verp call to the SMTP server is to replace MAIL FROM:<sender@domain> by MAIL FROM:<sender@domain> XVERP=+= I don't know what this would do for performance with postfix (relative to having Mailman do the VERPing), but it would certainly be interesting to find out. Can you modify your code so as to generate this modified string, and see what happens?
There's already a patch that adds the required options to do this into SMTPDirect.py
http://sourceforge.net/tracker/index.php?func=detail&aid=869638&group_id=103&atid=300103
I've got it on a production system here, and it's working great. Though the largest list it's got is just 11000 addresses, which isn't enough to really stress it.
-- Colin Palmer <mailmandev@waikato.ac.nz> University of Waikato
Fil wrote:
For example, it might be faster/lower overall load on the server if we had the MTA do the VERPing for us -- we're pretty sure that's supported by some MTAs (e.g., at least some versions of Exim), and we know it's faster for at least some of them (e.g., Exim).
Sorry, for postfix indeed it might be much easier to do it this way: http://www.postfix.org/VERP_README.html#smtp
the only change between non-verp and verp call to the SMTP server is to replace MAIL FROM:<sender@domain> by MAIL FROM:<sender@domain> XVERP=+=
-- Fil
I've been doing Postfix VERPing for years. It's a really small change to the sendmail invoker (it already had a variable to hold 'options' or some such.)
On Tue, 2004-07-13 at 12:52, Brad Knowles wrote:
At 11:08 AM +0200 2004-07-13, Fil wrote:
I hope this feedback can be useful, if you have scalability in mind for MM3.
This is really more of an MTA limitation, although there might be some things we can do to try to work around it with Mailman. For example, it might be faster/lower overall load on the server if we had the MTA do the VERPing for us -- we're pretty sure that's supported by some MTAs (e.g., at least some versions of Exim), and we know it's faster for at least some of them (e.g., Exim).
If you are doing VERP in MM then you send one message per recipient from MM to your MTA. Thats the think that will cause problems on many systems because you will need almost as much spool space as the message multiplied by the recipient count (almost as much because some of the early messages will have left the box by the time the last ones come in).
I guess that the awful performance in this case is probably down to something like the MTA doing verification on sender/recipient inline, which will slaughter performance.
We are very very shortly going to convert exim.org to a new platform with new Mailman installation, which I intend to do VERP deliveries on. I can use that box, when in use, to test VERP on MM against VERP within exim, and I'll write up my results. However I am looking at 2K users - not 150K, and it maybe that there are scaling problem
As a blatant and unwarranted plug, the Exim/Mailman HOWTO gives information on using VERP with Exim http://www.exim.org/howto/mailman21.html
Cheers Nigel.
[At an exim conference :-) ]
-- [ Nigel Metheringham Nigel.Metheringham@InTechnology.co.uk ] [ - Comments in this message are my own and not ITO opinion/policy - ]
Here's the smtp log line: Jul 13 02:05:22 2004 (435) <20040712150834.DCA153680BD@alan.rezo.net> smtp for 152991 recips, completed in 31782.241 seconds .../... In any case, 2 seconds per VERP message for such a big list is too costly,
Someone kindly told me this makes only 0.2s /message, not 2 seconds
-- Fil
participants (7)
-
Brad Knowles -
Colin Palmer -
Dan Mick -
Darrell Fuhriman -
Fil -
Nigel Metheringham -
Somuchfun