Bounce processing performance, pending.pck possible bug/redesign
I'm currently running Mailman 2.1.1 I recently completed splitting my large 190,000 member list into 72 sublists for performance reasons (primarily bounce processing).
I split the lists leaving bounce disabled addresses in the mix to be able to measure performance of the new setup.
My first mailing to my 72 sublists generated over 15,000 bounces, however the performance wasn't what I expected.
The first few bounces were processed at over 8 per second. However that slowed to 5 seconds PER BOUNCE near the end.
After investigation I noticed that ~mailman/data/pending.pck had grown to almost 2 megabytes in size.
Even though I have my bounce threshold set to 5.0 and NO addresses were disabled because of bouncing, pending.pck contained over 15,000 "re-enable" entries along with generated confirmation strings.
I double checked the bounce information was correctly registered by dumping the config.pck for the mailing list. None of these addresses were disabled and the bounce information is correct.
I looked at the code in Bouncer.py at the "registerbounce" routine and this behavior appears to be by design. Am I missing something here or is this a bug?
At 13:45 26/04/2003, John Distler wrote:
I'm currently running Mailman 2.1.1 I recently completed splitting my large 190,000 member list into 72 sublists for performance reasons (primarily bounce processing).
I split the lists leaving bounce disabled addresses in the mix to be able to measure performance of the new setup.
My first mailing to my 72 sublists generated over 15,000 bounces, however the performance wasn't what I expected.
The first few bounces were processed at over 8 per second. However that slowed to 5 seconds PER BOUNCE near the end.
After investigation I noticed that ~mailman/data/pending.pck had grown to almost 2 megabytes in size.
Even though I have my bounce threshold set to 5.0 and NO addresses were disabled because of bouncing, pending.pck contained over 15,000 "re-enable" entries along with generated confirmation strings.
With a threshold of 5.0, it is going to require multiple bounces for each mail address to get it disabled. MM needs to keep records to do this.
If you have a 'dirty' membership list, why not drop the threshhold to less than one e.g. 0.5, so that the next mailing disables the bouncing subscribers immediately; that is assuming that the returning addresses matches the subscribed addresses and you are not using VERP.
I double checked the bounce information was correctly registered by dumping the config.pck for the mailing list. None of these addresses were disabled and the bounce information is correct.
I looked at the code in Bouncer.py at the "registerbounce" routine and this behavior appears to be by design. Am I missing something here or is this a bug?
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
Thanks for the reply.
I should have been more clear in my original message:
Only when a subscriber exceeds the bounce threshold (and gets disabled) should a record be added to the pending.pck file. From that point on "You are disabled messages" would be sent to the subscriber with a confirmation string.
It appears that Bouncer.py currently makes an entry in pending.pck for the very FIRST bounce even though the bounce disable threshold is set MUCH higher than one for the list.
It is my understanding that the bounce date and count information is stored in the config.pck database.
It appears than the extra processing of the pending.pck database may be increasing the bounce processing time significantly and possibly unnecessarily.
Richard, I also want to thank you for your earlier help with umbrella lists it was invaluable.
It appears that Bouncer.py currently makes an entry in pending.pck for the very FIRST bounce even though the bounce disable threshold is set MUCH higher than one for the list.
I would agree.
It is my understanding that the bounce date and count information is stored in the config.pck database.
Yes, and this choice is pretty much impossible to understand: it increases the risk of corruption of the config.pck file, and it makes the BounceRunner lock the config.pck file for ages when it needs just lock the "bouncers" part of that file.
It appears than the extra processing of the pending.pck database may be increasing the bounce processing time significantly and possibly unnecessarily.
In my view it would be better to have a seperate bounce.pck file
-- Fil
Yes, a redesign would ultimately be the solution.
BUT, Is the current behavior a bug (writing a confirmation string on the first bounce) that should be easily fixed for better performance or am I not understanding the logic?
This is for Mailman 2.1 or later
Create a directory as mailman.mailman called ~mailman/lists/your-list-name/language-of-choice/
where your-list-name is the name of your list and language-of-choice would be "en" for english "de" for german, etc.
Copy the file subscribeack.txt
from
~mailman/templates/language-of-choice/
to
~mailman/lists/your-list-name/language-of-choice/
Edit subscribeack.txt with your favorite text editor so it contains this on the first line:
%(welcome)s
This will leave only the contents of the welcome message that you entered in the web ui
Cheers!
participants (3)
-
Fil -
John Distler -
Richard Barrett