Suggestions about Mailman bounce processing
I'm not a subscriber to this list, so please include me superticker2{at}iastate.edu on your cc: line.
I have two suggestions I would like to make concerning Mailman bounce processing:
- All bounces from either a Mass Subscribe or Mass Invite operation should be unconditionally and immediately sent to the list owner.
There can be many typos in these Mass Subscribe/Invite lists, and the list owner needs to correct these typos _immediately_. In addition, some of these mass subscription lists come from paper sign-up lists by foreign students where the English alphabet is new to them. Trying to decipher their email addresses can result in 10% typos and bounces, which need immediate corrections.
- All bounces should be immediately directed to the list owner if bounce processing is switched off.
For small lists, it makes the most sense for the list owner to handle bounces manually. Often he'll know the recipients personally and their new email addresses, which he can fix himself when they bounce. As it stand today, there could be 100s of bad addresses on an email list without anyone's knowledge if bounce processing is off; this is bad design.
Superticker2 (Mark) writes:
-1
You seem to be confusing your personal (including job-related) idiosyncracies with universal requirements. Mailman is used by many different people with many different needs. For example, in my use case, I don't want to see such bounces; I never hand-type addresses, but rather use cut and paste from mails sent to me personally or one of my administrative aliases (incorrectly; my target users are sophisticated users, and are expected to handle subscriptions themselves). If that bounces, there is *nothing* I can do to correct it (as it turns out, in over ten years of this practice AFAIK I have never made a mistake in the process of copy/paste; doesn't mean it won't happen tomorrow but I'm not going to worry about it!), and ordinary bounce processing works fine for cleaning the list. (It takes about a month for an address to be removed, but during that period it is disabled and uses negligible resources.)
It is possible to set up bounce processing so that your requirements are met (bounce_processing = 1, bounce_score_threshold = 0.0, bounce_you_are_disabled_warnings = 0, bounce_notify_owner_on_removal = 1 -- granted, the user is removed from the list ;-).
AFAIK those conditions apply to bounces from admin-originated mail as well as ordinary list traffic. If not, I agree that it should be made possible, and probably there should be an option to have different options applied to bounces caused by admin-originated mail, although if it's that important that admin-originated mail be checked, you probably want to know immediately if a subscriber goes AWOL, too.
-1
Bounce processing is on by default. If an owner switches it off, presumably they know what they're doing.
For small lists, it makes the most sense for the list owner to handle bounces manually.
Not if the list address is visible from the "InterSewer". Eventually either the -bounce address will be used as a Sender by a spammer or it will end up on the spammer's To list. Enough of these will get through to make the list owner's task not merely a chore, but a disgusting chore.
Not to mention that a large fraction of bounces are actually due to transient causes (host goes offline on a holiday weekend, for example). And in many use cases the most common cause of bounces is "undeliverable; over-quota".
Doesn't bounce_processing = 1, bounce_unrecognized_goes_to_list_owner = 1 do exactly what you want? If it doesn't, it should be fixed to do so. Hm ... see below.
Looks like an oversight to me. The people I know just don't run without automatic bounce processing in this day and age if they want to do useful work besides manually processing bounces, so nobody has noticed the bug. At l.224 of Mailman/Queue/BounceRunner.py, this change should allow you to configure things so that you get all the bounces you want (and if the addresses of your lists are visible from the InterNet, quite a few you don't want):
# List isn't doing bounce processing?
if not mlist.bounce_processing:
maybe_forward(mlist, msg) return
(This is Mailman 2.10, my host is a bit behind the times, so the line number may be a bit off.)
Let me respond to each of your comments:
- We are a University campus group and the University currently provides a Mailman service to us. During our bi-annual university Club Fair, we can
collect 200 email addresses--on paper--from prospective students/staff wanting on our list. Many are Asian students, and their English handwriting is poor at best, but we even have trouble reading American email addresses. We guess on the bad handwriting and 10% (20 of them) bounce. We can correct those write away and get most of them working on the second try (or do an LDAP lookup on the problem ones).
Using a paper sign-up list for our booth at the Club Fair is _not_ ideal, but it's all we have available in this theater. It's nice you don't have too use a paper sign-up list like we do.
I appreciate your suggestion to set bounce_processing=1, bounce_score_threshold=0, but wouldn't this would cause "existing addresses" with scores at "2" to be removed immediately, which we don't want to do. We want the existing addresses to remain until their bounce score reaches 2.5.
Or are you saying the existing scores will be unaffected by temporarily setting bounce_processing=1,threshold=0 for just the Mass Subscribe operation? If so, maybe you're on to something. But wouldn't it be prudent to make this the default behavior for all Mass Subscribes/Invites? Isn't it better to correct typos and such right away 100% of the time?
- I agree with your comments. But I currently run a 1500 subscriber announcement-only Majordomo list for a monthly newsletter with very good success processing my own bounces manually. I get about 8 to 10 "hard" bounces per newsletter, which I can handle. Yes, the soft bounces (over quota) are annoying.
But for a discussion list (not announcement-only) with messages everyday, you would be crazy not to run automatic bounce processing. I totally agree there.
The big advantage with manual bounce processing (small, monthly, announcement-only list) is you can just fix the bouncing problems yourself if you know everyone on the list. That would be the _only_ reason to disable bounce processing, that is, for manual bounce processing.
Thanks for listening and your suggestion for #1. I'll try that, but I still maintain that for any Mass Operation, you should be informed immediately of the typos--no special temporary settings required. Happy computing to you, Mark
*********** REPLY SEPARATOR ***********
On 12/27/2010 at 10:11 PM Stephen J. Turnbull wrote:
Executive summary:
It seems to me that your best bet is to modify Mailman as I suggest, and at times when you have a lot of paper signups, switch off "bounce processing" temporarily, and switch on (preferably it wasn't off in the first place) "bounce unrecognized goes to list owner". If modification scares you, surely somebody around is willing to check your work.
Of course you may not have access to the Mailman installation, but I would think that the ssite admin would at least consider this simple change, especially if Mark or Barry wants it for their next release(s), since it at worst ends up sending some extra mail to list owners who have volunteered for admin work in some sense, and only those who switch off bounce processing.
Superticker2 (Mark) writes:
Sure, been there, done that, and the settings you propose make sense for you. They'd just really annoy a lot of people, to no purpose whatsoever, as defaults.
Surely somebody has a cheap old notebook that you could let people type on at the booth? It doesn't have to be hooked to the net there. (That's a suggestion; I understand it may not work for you for various reasons.) Or have them mail you from their cellphones. For people who wish to use their cellphones as list address, set up a cellphone with the subscribe address as the prmary address of the owner, temporarily and exchange addresses. (Ditto the previous caveat.)
I don't think it causes them to be removed immediately, but it would cause those with scores at 1.5 to get removed on the next bounce, so it's probably out for you.
If so, maybe you're on to something. But wouldn't it be prudent to make this the default behavior for all Mass Subscribes/Invites?
Not default, for the reasons given.
Isn't it better to correct typos and such right away 100% of the time?
Remember, I don't make typos of that kind, though I'm human enough in all other contexts. I suspect that most people using mass subscribe/ mass invite are not transcribing from paper, and if so, they probably don't make typos very often either. People do make such typos, though, and if they do, there's little you can do about it even if you want to. It's a bad idea to try to guess a correct address from an incorrect one, especially if you don't know the spelling rules of the language in question.
Then AFAICS you agree it should not be default. ;-)
Sure, I'd want that too. But the computer can't tell if they *are* the list admin's typos. You have a special case which generates a lot of correctable typos, and in my (limited but non-nil) experience, it's a rare one, and getting more so all the time. So it's not a great default.
Note that the final decision is Marks (v2.x) or Barry's (v3)q, of course. But I tend to think they'll agree with me.
What might be simplest solution here would be to temporarily (say for 40 minutes) switch bounce processing off just _before_ a Mass Subscribe/Invite operation so that all bounces are copied to the list owner. Ideally, we only care about the bounces for the last Mass Operation, but the implementation would be simplest to just copy all of them to the list owner while bounce processing is temporarily disabled.
Seeing the bounces may also help with debugging/optimizing the bounce processing threshold and such.
I can't imagine why a list owner would want to switch bounce processing off without seeing what's bouncing? And one reason we want to switch from Majordomo to Mailman is to use its automatic bounce processing (except on Mass Subscribes/Invites). :)
If there's a service or utility that can verify a list of email addresses (prior to a mass operation), please suggest it.
Thanks again for listening and your suggestions, Mark
*********** REPLY SEPARATOR ***********
On 12/28/2010 at 12:27 AM Stephen J. Turnbull wrote:
Superticker2 (Mark) wrote:
If there's a service or utility that can verify a list of email addresses (prior to a mass operation), please suggest it.
There are some. See for example <http://www.tienhuis.nl/php-email-address-validation-with-verify-probe> or try this google search <http://www.google.com/#q=verify+email+address>.
There are problems with trying to do address verification without actually sending mail. The obvious way to verify an address is to send an SMTP VRFY command to the MX of the domain of the address, but these days most SMTP servers either don't implement VRFY or won't accept VRFY from untrusted sources.
The alternative is to begin an SMTP dialog with the MX and send MAIL FROM and RCPT TO commands. As long as you don't send DATA, no mail will actually be sent, but this is not reliable. If the MX rejects the RCPT, you may know from the status that the address is bad, but some MXs will accept an invalid RCPT address and not respond with failure until after DATA, and you can't continue to that point if you don't want to actually send mail.
For example, consider the following two SMTP sessions
$ telnet a.mx.mail.yahoo.com 25 Trying 67.195.168.31... Connected to a.mx.mail.yahoo.com (67.195.168.31). Escape character is '^]'. 220 mta178.mail.ac4.yahoo.com ESMTP YSmtp service ready HELO msapiro.net 250 mta178.mail.ac4.yahoo.com MAIL FROM: <mark@msapiro.net> 250 sender <mark@msapiro.net> ok RCPT TO: <totally_bogus_mark_sapiro@yahoo.com> 250 recipient <totally_bogus_mark_sapiro@yahoo.com> ok DATA 354 go ahead . 554 Transaction failed: empty DATA not allowed Connection closed by foreign host.
and
$ telnet a.mx.mail.yahoo.com 25 Trying 67.195.168.31... Connected to a.mx.mail.yahoo.com (67.195.168.31). Escape character is '^]'. 220 mta112.mail.ac4.yahoo.com ESMTP YSmtp service ready HELO msapiro.net 250 mta112.mail.ac4.yahoo.com MAIL FROM: <mark@msapiro.net> 250 sender <mark@msapiro.net> ok RCPT TO: <totally_bogus_mark_sapiro@yahoo.com> 250 recipient <totally_bogus_mark_sapiro@yahoo.com> ok DATA 354 go ahead From: me To: you
body . 554 delivery error: dd This user doesn't have a yahoo.com account (totally_bogus_mark_sapiro@yahoo.com) [0] - mta112.mail.ac4.yahoo.com Connection closed by foreign host.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Trying to verify the email address prior to the Mass Subscribe/Invite is the easiest solution. I'm going to look into that first. Thanks for all the tips below.
But as pointed out below, because of spammers trying to randomly guess email addresses, I think today's mail servers don't always respond with reliable answers. This makes address verification problematic, although it can't hurt to try finding some of bad addresses that way.
Sorry, I haven't learned PHP as yet. My most fluent programming languages are C, C++, G (LabView), and MatLab.
Happy computing, Mark
*********** REPLY SEPARATOR ***********
On 12/28/2010 at 3:29 PM Mark Sapiro wrote:
"Superticker2 (Mark)" schrieb:
If there's a service or utility that can verify a list of email addresses (prior to a mass operation), please suggest it.
<http://th-h.de/download/checkmail-0.4.tar.gz> is a Perl script to check the validity of a mail address (by trying a SMTP dialog up to RCPT TO).
Regards, -thh
On Dec 29, 2010, at 12:50 AM, Thomas Hochstein wrote:
FWIW, in Mailman 3, email validation is a "utility" in the ZCA[*] sense of the term, so in theory it should be fairly easily pluggable. The current algorithm is not much different than the Mailman 2 checks.
-Barry
[*] Zope Component Architecture
Stephen J. Turnbull wrote:
Stephen is correct. Those with non-stale scores >= 1 will not be removed immediately, but they will be removed the next time cron/disabled runs (default 09:00 daily) if the threshold is not raised before that.
It seems pretty safe to do this. Sometime after cron/disabled runs reduce the threshold to <= 1, make sure that bounce_notify_owner_on_disable is Yes and do the mass subscribe. Since no list members are mailed in this process, they won't bounce. After allowing time for bounces to be returned and processed (bounces are queued in Mailman and only processed at 15 minute intervals), raise the threshold to the original value.
Note however that this will only work for mass subscribes. For invitations, the returned bounces will be for addresses which aren't list members and these are always ignored.
Note that the final decision is Marks (v2.x) or Barry's (v3)q, of course. But I tend to think they'll agree with me.
FWIW, I do agree.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
This is VERY interesting. I really appreciate this suggestion and may try it after confirming that disables occur at 9:00AM local time. Thanks for your insights!
Another thought I had would be to setup a list-owner web page for each emailing list that would run a grep script on the Mailman "Bounces Log" that would (1) show recent (last hour) bounce activity for (2) that specific list following a Mass Subscribe/Invite operation. I don't have access to the Mailman server itself, but if such a web page could give me a window into the Bounces Log for my specific list, that would be all I need. I just need to identify which addresses are bouncing.
*********** REPLY SEPARATOR ***********
On 12/28/2010 at 2:54 PM Mark Sapiro wrote:
participants (5)
-
Barry Warsaw
-
Mark Sapiro
-
Stephen J. Turnbull
-
Superticker2 (Mark)
-
Thomas Hochstein