Cron job to reset member bounce value
Can anyone tell me where the actual member bounce value is stored? I want to setup a cron job to reset the member bounce value to 0 for certain members whose mailboxes are only active when they have a satellite overhead to give them internet connectivity (about 8-10 hours each day). They keep getting disabled because their bounce value breaks the threshold value but this cannot be avoided due to their location and lack of internet. Thank you for your time and knowledge.
On 12/16/19 9:55 AM, Ethan Rudnitsky wrote:
The score is in the member's _BounceInfo object defined in Mailman/Bouncer.py. This object is retrieved and set by the list's getBounceInfo() and setBounceInfo() methods defined in the MemberAdaptor (Mailman/OldStyleMemberships.py by default).
However, See the script at <https://www.msapiro.net/scripts/clear_bounce_info.py> (mirrored at <https://fog.ccsf.edu/~msapiro/scripts/clear_bounce_info.py>) which will do what you want.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/16/2019 9:55 AM, Ethan Rudnitsky wrote:
Sounds like a work-around to the real problem- that one of the MTA in the delivery chain is not properly retrying the message. In general, most MTAs are configured to retry for at least 24 hours and sometimes as long as 5 days before returning a fatal bounce, so if a destination server is off-line, the sending one will try again in an hour or so.
Which error code(s) are you seeing? Some 400 codes are designed to be retried later whereas 500's are immediately fatal.
Later,
z!
Mark and Carl,
Thank you both for your replies. So there are actually 2 issues causing bounces here... the first is the mailbox being offline at times, and the second is that it is getting messages bounced because of message size being over the limit. A few of these same accounts have strict 50KB message limits because they get their connectivity via Iridium modems, so when someone sends a message greater than 50KB it is bouncing. I have just utilized one of Mark's scripts and created a cron job in the crontab.in file at /usr/lib/mailman/cron that is formatted as follows: 0 8 * * * mailman /usr/lib/mailman/bin/withlist -a -r reset_bounce -- -u manager@xxxxxxxx.xxx.xxx -u comms@xxxxxxxx.xxx.xxx -u cargo@xxxxxxxx.xxx.xxx
I then restarted the mailman service. Does this to be properly formatted for targeting only the 3 specific member accounts I want?
Thank you both for your time!
-Ethan
On Tue, Dec 17, 2019 at 10:39 AM Carl Zwanzig <cpz@tuunq.com> wrote:
On 12/17/2019 9:45 AM, Ethan Rudnitsky wrote:
Sounds like the -server- (aka 'MTA') is offline, and those messages should be retried; if they're not being retried, something is generally wrong and resetting the bounce counts is only masking that problem.
Another option is to create a size-limited list and add that to the main list. Or just add a size limit to the existing list.... unless you're sending pictures or large files, most things can be said in under 50kb :) (and IMHO there are better ways of distributing larger files than email).
Later,
z!
On 12/17/19 9:45 AM, Ethan Rudnitsky wrote:
Carl raises some good points, but to answer your specific question, I don't know what if anything your distro's Mailman package does with /usr/lib/mailman/cron/crontab.in, but if it copies it automagically to /etc/cron.d/mailman, then putting the command there is OK.
The bottom line is if your Mailman cron jobs are in a system crontab such as /etc/cron.d/mailman, you want the line
0 8 * * * mailman /usr/lib/mailman/bin/withlist -a -r reset_bounce -- -u manager@xxxxxxxx.xxx.xxx -u comms@xxxxxxxx.xxx.xxx -u cargo@xxxxxxxx.xxx.xxx
in that file. If your Mailman cron jobs are in the Mailman user's crontab /var/spool/cron/crontabs/mailman, you need to omit the username as in
0 8 * * * /usr/lib/mailman/bin/withlist -a -r reset_bounce -- -u manager@xxxxxxxx.xxx.xxx -u comms@xxxxxxxx.xxx.xxx -u cargo@xxxxxxxx.xxx.xxx
You also need to have copied the script to /usr/lib/mailman/bin/reset_bounce.py
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark and Carl,
Thanks again for the replies.
Carl, the issue with the server/MTA bring offline is a known and unavoidable issue because of lack of satellite availability overhead for most of the day unfortunately. Regarding the mail size limit, those messages are ones that are really quite small (talking like 500kb) but still go over the 50kb limit of the mailbox. It is definitely a system that we are looking to replace but the geographic location and limited satellite coverage leave us with few options.
Mark, I have confirmed that the crontab.in does automatically copy /etc/cron.d/mailman, and /var/spool/cron/ is empty. I did also initially copy the reset_bounce.py to the directory you mentioned, so thank you for your diligence in checking, sir!
I guess we'll see how it behaves and how it goes. Thanks again; I really appreciate it.
-Ethan
On Tue, Dec 17, 2019 at 11:14 AM Mark Sapiro <mark@msapiro.net> wrote:
On 12/17/2019 10:24 AM, Ethan Rudnitsky wrote:
That is an entirely avoidable issue. When a destination MTA is offline/busy (which happens all the time and is accounted-for in SMTP), the sending MTA usually queues the messages and retries in 15/30/120 minutes, and usually keeps that up for at least 24 hours. If the sending MTA is giving up immediately and returning a failure, IMHO it's misconfigured (knowing the exact error code returned is important, here).
That said, the operator(s) of that server may not be interested in fixing the problem.
Do the satellite-served list members receive every message eventually or are they content to only receive ones sent during their connectivity window? If the -do- get all the messages eventually then they are being retried and the temporary failure is being counted as a bounce.
Later,
z!
Carl, my apologies as I see what you are saying now. The server is indeed retrying instead of giving up right away, so they do receive the messages eventually but the retries are being counted as a bounce.
Mark, when I run your get_bounce_info script I do get an output of the current score for each member in each list, but when I manually run "./withlist -a -r reset_bounce --" it returns "Importing reset_bounce..." [newline] "Running reset_bounce.reset_bounce()... and it then lists all of the lists on each line as it goes through them and appends each with an (unlocked) and then says Finalizing. However, if I run the get_bounce_info again, nothing is reset to 0 bounces, it lists the same thing as before. So I guess I need clarification, will this only reset bounce counts for members who are already/currently locked, or is it supposed to reset the count back to 0 even if they are unlocked and below the threshold?
Thank you, sir.
On Tue, Dec 17, 2019 at 11:34 AM Carl Zwanzig <cpz@tuunq.com> wrote:
On 12/17/19 1:28 PM, Ethan Rudnitsky wrote:
If the message delayed DSNs or whatever they are are being scored as bounces by Mailman, this is a bug. Please send me one of the bounce messages you receive that increments the score or disables the member. You get these by setting bounce_notify_owner_on_disable and if your Mailman is new enough (>=2.1.19) bounce_notify_owner_on_bounce_increment on the Bounce Processing page in the web admin UI.
Send the full, raw message as attached to the notice or the full, raw notice, and I will try to update the recognizer to recognize it as non-fatal.
You seem to have picked the wrong script, and I didn't notice in my follow-up. The reset_bounce script will re-enable delivery for all members with delivery already disabled by bounce. It will not reset scores for members not yet disabled. This is probably not what you want. The script you want is one that resets the current bounce info and score for members with current bounce info but not yet disabled. That script is <https://www.msapiro.net/scripts/clear_bounce_info.py> (mirrored at <https://fog.ccsf.edu/~msapiro/scripts/clear_bounce_info.py>.
All of the
Importing reset_bounce... Running reset_bounce.reset_bounce()... and it then lists all of the lists on each line as it goes through them and appends each with an (unlocked) and then says Finalizing
output comes from withlist as it imports the script and runs it against the lists.
The scripts themselves produce no output unless you specify the -v option (after the --) in which case they just print a count of the number of users affected.
The -v output for clear_bounce info is
List <listname>: Cleared bounce info for <nn> members.
and for reset_bounce it is
List <listname>: Reset <nn> bouncing members.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/16/19 9:55 AM, Ethan Rudnitsky wrote:
The score is in the member's _BounceInfo object defined in Mailman/Bouncer.py. This object is retrieved and set by the list's getBounceInfo() and setBounceInfo() methods defined in the MemberAdaptor (Mailman/OldStyleMemberships.py by default).
However, See the script at <https://www.msapiro.net/scripts/clear_bounce_info.py> (mirrored at <https://fog.ccsf.edu/~msapiro/scripts/clear_bounce_info.py>) which will do what you want.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/16/2019 9:55 AM, Ethan Rudnitsky wrote:
Sounds like a work-around to the real problem- that one of the MTA in the delivery chain is not properly retrying the message. In general, most MTAs are configured to retry for at least 24 hours and sometimes as long as 5 days before returning a fatal bounce, so if a destination server is off-line, the sending one will try again in an hour or so.
Which error code(s) are you seeing? Some 400 codes are designed to be retried later whereas 500's are immediately fatal.
Later,
z!
Mark and Carl,
Thank you both for your replies. So there are actually 2 issues causing bounces here... the first is the mailbox being offline at times, and the second is that it is getting messages bounced because of message size being over the limit. A few of these same accounts have strict 50KB message limits because they get their connectivity via Iridium modems, so when someone sends a message greater than 50KB it is bouncing. I have just utilized one of Mark's scripts and created a cron job in the crontab.in file at /usr/lib/mailman/cron that is formatted as follows: 0 8 * * * mailman /usr/lib/mailman/bin/withlist -a -r reset_bounce -- -u manager@xxxxxxxx.xxx.xxx -u comms@xxxxxxxx.xxx.xxx -u cargo@xxxxxxxx.xxx.xxx
I then restarted the mailman service. Does this to be properly formatted for targeting only the 3 specific member accounts I want?
Thank you both for your time!
-Ethan
On Tue, Dec 17, 2019 at 10:39 AM Carl Zwanzig <cpz@tuunq.com> wrote:
On 12/17/2019 9:45 AM, Ethan Rudnitsky wrote:
Sounds like the -server- (aka 'MTA') is offline, and those messages should be retried; if they're not being retried, something is generally wrong and resetting the bounce counts is only masking that problem.
Another option is to create a size-limited list and add that to the main list. Or just add a size limit to the existing list.... unless you're sending pictures or large files, most things can be said in under 50kb :) (and IMHO there are better ways of distributing larger files than email).
Later,
z!
On 12/17/19 9:45 AM, Ethan Rudnitsky wrote:
Carl raises some good points, but to answer your specific question, I don't know what if anything your distro's Mailman package does with /usr/lib/mailman/cron/crontab.in, but if it copies it automagically to /etc/cron.d/mailman, then putting the command there is OK.
The bottom line is if your Mailman cron jobs are in a system crontab such as /etc/cron.d/mailman, you want the line
0 8 * * * mailman /usr/lib/mailman/bin/withlist -a -r reset_bounce -- -u manager@xxxxxxxx.xxx.xxx -u comms@xxxxxxxx.xxx.xxx -u cargo@xxxxxxxx.xxx.xxx
in that file. If your Mailman cron jobs are in the Mailman user's crontab /var/spool/cron/crontabs/mailman, you need to omit the username as in
0 8 * * * /usr/lib/mailman/bin/withlist -a -r reset_bounce -- -u manager@xxxxxxxx.xxx.xxx -u comms@xxxxxxxx.xxx.xxx -u cargo@xxxxxxxx.xxx.xxx
You also need to have copied the script to /usr/lib/mailman/bin/reset_bounce.py
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark and Carl,
Thanks again for the replies.
Carl, the issue with the server/MTA bring offline is a known and unavoidable issue because of lack of satellite availability overhead for most of the day unfortunately. Regarding the mail size limit, those messages are ones that are really quite small (talking like 500kb) but still go over the 50kb limit of the mailbox. It is definitely a system that we are looking to replace but the geographic location and limited satellite coverage leave us with few options.
Mark, I have confirmed that the crontab.in does automatically copy /etc/cron.d/mailman, and /var/spool/cron/ is empty. I did also initially copy the reset_bounce.py to the directory you mentioned, so thank you for your diligence in checking, sir!
I guess we'll see how it behaves and how it goes. Thanks again; I really appreciate it.
-Ethan
On Tue, Dec 17, 2019 at 11:14 AM Mark Sapiro <mark@msapiro.net> wrote:
On 12/17/2019 10:24 AM, Ethan Rudnitsky wrote:
That is an entirely avoidable issue. When a destination MTA is offline/busy (which happens all the time and is accounted-for in SMTP), the sending MTA usually queues the messages and retries in 15/30/120 minutes, and usually keeps that up for at least 24 hours. If the sending MTA is giving up immediately and returning a failure, IMHO it's misconfigured (knowing the exact error code returned is important, here).
That said, the operator(s) of that server may not be interested in fixing the problem.
Do the satellite-served list members receive every message eventually or are they content to only receive ones sent during their connectivity window? If the -do- get all the messages eventually then they are being retried and the temporary failure is being counted as a bounce.
Later,
z!
Carl, my apologies as I see what you are saying now. The server is indeed retrying instead of giving up right away, so they do receive the messages eventually but the retries are being counted as a bounce.
Mark, when I run your get_bounce_info script I do get an output of the current score for each member in each list, but when I manually run "./withlist -a -r reset_bounce --" it returns "Importing reset_bounce..." [newline] "Running reset_bounce.reset_bounce()... and it then lists all of the lists on each line as it goes through them and appends each with an (unlocked) and then says Finalizing. However, if I run the get_bounce_info again, nothing is reset to 0 bounces, it lists the same thing as before. So I guess I need clarification, will this only reset bounce counts for members who are already/currently locked, or is it supposed to reset the count back to 0 even if they are unlocked and below the threshold?
Thank you, sir.
On Tue, Dec 17, 2019 at 11:34 AM Carl Zwanzig <cpz@tuunq.com> wrote:
On 12/17/19 1:28 PM, Ethan Rudnitsky wrote:
If the message delayed DSNs or whatever they are are being scored as bounces by Mailman, this is a bug. Please send me one of the bounce messages you receive that increments the score or disables the member. You get these by setting bounce_notify_owner_on_disable and if your Mailman is new enough (>=2.1.19) bounce_notify_owner_on_bounce_increment on the Bounce Processing page in the web admin UI.
Send the full, raw message as attached to the notice or the full, raw notice, and I will try to update the recognizer to recognize it as non-fatal.
You seem to have picked the wrong script, and I didn't notice in my follow-up. The reset_bounce script will re-enable delivery for all members with delivery already disabled by bounce. It will not reset scores for members not yet disabled. This is probably not what you want. The script you want is one that resets the current bounce info and score for members with current bounce info but not yet disabled. That script is <https://www.msapiro.net/scripts/clear_bounce_info.py> (mirrored at <https://fog.ccsf.edu/~msapiro/scripts/clear_bounce_info.py>.
All of the
Importing reset_bounce... Running reset_bounce.reset_bounce()... and it then lists all of the lists on each line as it goes through them and appends each with an (unlocked) and then says Finalizing
output comes from withlist as it imports the script and runs it against the lists.
The scripts themselves produce no output unless you specify the -v option (after the --) in which case they just print a count of the number of users affected.
The -v output for clear_bounce info is
List <listname>: Cleared bounce info for <nn> members.
and for reset_bounce it is
List <listname>: Reset <nn> bouncing members.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Carl Zwanzig -
Ethan Rudnitsky -
Mark Sapiro