Fwd: Bounce action notification
this is a bug in the bounce processing for Mailman 2.1
Begin forwarded message:
From: mailman@plaidworks.com Date: Tue Feb 25, 2003 12:58:07 AM US/Pacific To: sharks-owner@plaidworks.com Subject: Bounce action notification
This is a Mailman mailing list bounce action notice:
List: sharks Member: xxxx@hotmail.com Action: Subscription disabled. Reason: Excessive or fatal bounces.The triggering bounce notice is attached below.
Questions? Contact the Mailman site administrator at mailman@plaidworks.com.
From: postmaster@mail.hotmail.com Date: Tue Feb 25, 2003 12:50:14 AM US/Pacific To: sharks-bounces+xxxx=hotmail.com@plaidworks.com Subject: Delivery Status Notification (Delay)
This is an automatically generated Delivery Status Notification.
THIS IS A WARNING MESSAGE ONLY.
YOU DO NOT NEED TO RESEND YOUR MESSAGE.
Delivery to the following recipients has been delayed.
xxxx@hotmail.comReporting-MTA: dns;mc10-s16.bay6.hotmail.com Received-From-MTA: dns;mc10-f40.bay6.hotmail.com Arrival-Date: Mon, 24 Feb 2003 12:49:41 -0800
Original-Recipient: Final-Recipient: rfc822;xxxx@hotmail.com Action: delayed Status: 4.4.7 Will-Retry-Until: Tue, 25 Feb 2003 12:49:41 -0800
-- Chuq Von Rospach, Architech chuqui@plaidworks.com -- http://www.plaidworks.com/chuqui/blog/
He doesn't have ulcers, but he's a carrier.
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> this is a bug in the bounce processing for Mailman 2.1What's the bug? That we're registering a hard bounce score for warning messages? That's a feature. :)
We "Precedence: list" on all our outgoing messages. MTAs that send delay warnings to such messages are broken.
-Barry
you go tell hotmail to fix itself, Barry. Seems to me, since it's fairly easy to parse the "this is a warning message" stuff, we ought to.
shrug. I guess.
On Tuesday, February 25, 2003, at 07:49 AM, Barry A. Warsaw wrote:
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> this is a bug in the bounce processing for Mailman 2.1What's the bug? That we're registering a hard bounce score for warning messages? That's a feature. :)
We "Precedence: list" on all our outgoing messages. MTAs that send delay warnings to such messages are broken.
-Barry
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
-- Chuq Von Rospach, Architech chuqui@plaidworks.com -- http://www.plaidworks.com/chuqui/blog/
No! No! Dead girl, OFF the table! -- Shrek
Chuq Von Rospach wrote:
you go tell hotmail to fix itself, Barry.
...or tell your users to drop that piece of crap.
yes yes, I know, millions on the Internet, elitist attitude, yadda yadda.
Seems to me, since it's fairly easy to parse the "this is a warning message" stuff, we ought to.
It is?
I've seen a *lot* of those sorts of messages. Every two-bit MTA admin in the world seems to want to customize his own.
On Tuesday, February 25, 2003, at 03:40 PM, Dan Mick wrote:
Chuq Von Rospach wrote:
you go tell hotmail to fix itself, Barry.
...or tell your users to drop that piece of crap.
yes yes, I know, millions on the Internet, elitist attitude, yadda yadda.
nah, it's even simpler: if I take a "me or him" attitude, chances are, they'll choose "him". Not a fight I want to take (I'm already having too much fun with AOL, as my blog will show. sigh. grump. I have better things to do), and not a fight I feel I can win.
hotmail is a 500 pound gorilla. I leave tilting at windmills to the youngsters out there, and prefer quiet co-existance.
Seems to me, since it's fairly easy to parse the "this is a warning message" stuff, we ought to.
It is?
I've seen a *lot* of those sorts of messages. Every two-bit MTA admin in the world seems to want to customize his own.
In the bounce system I've been writing, I'm up to about 99.46% resolution, including throwing out of soft bounces like this (having recently whacked 3 million or so bounces down to 20,000, I think I've got a pretty good test sample, too). and half of what I can't process are @$#@%W%% first class bombosities.
so yes, it's fairly easy. I doubt mailman can get QUITE to the level of processing I am, because I'm able to do some stuff you might not be able to get away with, but the answer to the general question is "i've already done it".
-- Chuq Von Rospach, Architech chuqui@plaidworks.com -- http://www.plaidworks.com/chuqui/blog/
He doesn't have ulcers, but he's a carrier.
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> so yes, it's fairly easy. I doubt mailman can get QUITE to
CVR> the level of processing I am, because I'm able to do some
CVR> stuff you might not be able to get away with, but the answer
CVR> to the general question is "i've already done it".Of course, Mailman can verp which takes the guesswork out of bounce processing. Even so, anything you can contribute to improve the non-verp bounce detection stuff would be appreciated.
-Barry
On Tuesday, February 25, 2003, at 07:25 PM, Barry A. Warsaw wrote:
Of course, Mailman can verp which takes the guesswork out of bounce processing. Even so, anything you can contribute to improve the non-verp bounce detection stuff would be appreciated.
that code's unfortunately not available.
But on a more general basis, I found a couple of techniques really helped me here, against all but the most obnoxious bounce returns (i.e. first class):
First, I now embed the user's email in the header in an X header, in two ways. One as the address, but also as a hashed string (MD5 hash, et al). I also use the hash in things like the unsubscribe links instead of the email address, because I've found the email address can cause issues of encoding on the way back, so you aren't always sure what to look for. the hash doesn't cause encoding issues.
then when stuff comes back, in most cases, the X- headers come back as part of the bounce, almost as reliable as VERP. And if I can't find that, I try to find the unsubscribe/info strings in the message or footer, and see if I can find the hash. if I do, I have an easy match. Nice thing about the hash it can't be deconstructed, and it also doesn't get cloaked by privacy things, so even if you run into sites trying to do "useful" stuff with email addresses, you can still get usable information.
the only place it fails is where people mailbot stuff and don't maintain any state from the original message (and those are relatively few, at least on a site/organizational level), and first class servers, but they're hopeless anyway.
-- Chuq Von Rospach, Architech chuqui@plaidworks.com -- http://www.plaidworks.com/chuqui/blog/
Very funny, Scotty. Now beam my clothes down here, will you?
Chuq Von Rospach wrote:
On Tuesday, February 25, 2003, at 07:25 PM, Barry A. Warsaw wrote:
Of course, Mailman can verp which takes the guesswork out of bounce processing. Even so, anything you can contribute to improve the non-verp bounce detection stuff would be appreciated.
that code's unfortunately not available.
But on a more general basis, I found a couple of techniques really helped me here, against all but the most obnoxious bounce returns (i.e. first class):
First, I now embed the user's email in the header in an X header, in two ways. One as the address, but also as a hashed string (MD5 hash, et al). I also use the hash in things like the unsubscribe links instead of the email address, because I've found the email address can cause issues of encoding on the way back, so you aren't always sure what to look for. the hash doesn't cause encoding issues.
The hash idea sounds interesting, but how do you manage to convert the hash back to an email address ?
I guess some simple email address mangling trick would give you the same effect, e.g. base64 encoding:
print 'mm(' + base64.encodestring('mal@lemburg.com') + ')' mm(bWFsQGxlbWJ1cmcuY29t)
These kind of encodings could even be used in URLs and survive text->HTML->text conversion.
-- Marc-Andre Lemburg eGenix.com
Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
M.-A. Lemburg wrote:
I guess some simple email address mangling trick would give you the same effect, e.g. base64 encoding:
print 'mm(' + base64.encodestring('mal@lemburg.com') + ')' mm(bWFsQGxlbWJ1cmcuY29t)
These kind of encodings could even be used in URLs and survive text->HTML->text conversion.
Here's some code and an example:
import base64, re
def encode_email_address(address, note=''): encoded = base64.encodestring(address + ' ' + note)[:-1] encoded = encoded.replace('=', '-') encoded = encoded.replace('+', '.') encoded = encoded.replace('/', '_') return 'mm(%s)' % encoded
def decode_email_address_part(mangled_address): mangled_address = mangled_address.replace('-', '=') mangled_address = mangled_address.replace('.', '+') mangled_address = mangled_address.replace('_', '/') decoded = base64.decodestring(mangled_address) return decoded.split()
message = """ From: postmaster@lemburg.com Date: Tue Feb 25, 2003 12:58:07 AM US/Pacific To: list-bounce@lists.egenix.com Subject: Bounce action notification
The following message bounced.
From: mailman@lemburg.com
To: info@egenix.com
...
_______________________________________________
%s
%s""" % (encode_email_address('mailman@lemburg.com', 'sender'), encode_email_address('info@egenix.com', 'rcpt'))
print 'Message:' print message print
parser = re.compile('mm\(([0-9a-zA-Z-._]+)\)')
def find_email_addresses(message, start=0, end=None): l = [] if end is None: end = len(message) while start < end: m = parser.search(message, start, end) if m is not None: address, note = decode_email_address_part(m.group(1)) l.append((note, address)) start = m.end() else: break return l
addrs = find_email_addresses(message) print 'Found these email address in message:' for addr in addrs: print ' ', addr
--
lemburg/tmp> python mangle_email_address.py Message:
From: postmaster@lemburg.com Date: Tue Feb 25, 2003 12:58:07 AM US/Pacific To: list-bounce@lists.egenix.com Subject: Bounce action notification
The following message bounced.
From: mailman@lemburg.com
To: info@egenix.com
...
_______________________________________________
mm(bWFpbG1hbkBsZW1idXJnLmNvbSBzZW5kZXI-)
mm(aW5mb0BlZ2VuaXguY29tIHJjcHQ-)Found these email address in message: ('sender', 'mailman@lemburg.com') ('rcpt', 'info@egenix.com')
-- Marc-Andre Lemburg eGenix.com
Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
On Thursday, February 27, 2003, at 02:33 AM, M.-A. Lemburg wrote:
The hash idea sounds interesting, but how do you manage to convert the hash back to an email address ?
you don't. Since I use a one-way hash (for various security reasons), you store the hash in the record with the email address, and then compare hashes. same general concept as an encrypted password.
One reason you do that is so that people can't script your CGIs to extract your subscription list programmatically, since the hash gives them no hook to hack your cgis with.
-- Chuq Von Rospach, Architech chuqui@plaidworks.com -- http://www.plaidworks.com/chuqui/blog/
Stress is when you wake up screaming and you realize you haven't fallen asleep yet.
Chuq Von Rospach wrote:
On Thursday, February 27, 2003, at 02:33 AM, M.-A. Lemburg wrote:
The hash idea sounds interesting, but how do you manage to convert the hash back to an email address ?
you don't. Since I use a one-way hash (for various security reasons), you store the hash in the record with the email address, and then compare hashes. same general concept as an encrypted password.
Ah, ok.
One reason you do that is so that people can't script your CGIs to extract your subscription list programmatically, since the hash gives them no hook to hack your cgis with.
So you have a third notion for identifying members... the hash address; in addition to LCE and KEY. Sounds like the data storage scheme needs an update :-)
-- Marc-Andre Lemburg eGenix.com
Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
On Tue, Feb 25, 2003 at 10:25:03PM -0500, Barry A. Warsaw wrote:
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> so yes, it's fairly easy. I doubt mailman can get QUITE to CVR> the level of processing I am, because I'm able to do some CVR> stuff you might not be able to get away with, but the answer CVR> to the general question is "i've already done it".Of course, Mailman can verp which takes the guesswork out of bounce processing. Even so, anything you can contribute to improve the non-verp bounce detection stuff would be appreciated.
-Barry
I thought VERP dealt with figuring out who a message was meant for, and that the problem here was distinguishing a warning message from a failure. Aren't those two separate issues?
"CVR" == Chuq Von Rospach <chuqui@plaidworks.com> writes:
CVR> you go tell hotmail to fix itself, Barry. Seems to me, since CVR> it's fairly easy to parse the "this is a warning message" CVR> stuff, we ought to. Dang, I forgot to read chapter 800 of the ISP handbook, "How to be a Friendly Gorilla". ;) Actually, this particular message appears to be a DSN, so it would be fairly trivial to ignore delayed actions. See line 48 of Mailman/Bouncers/DSN.py, and the commentary preceding that block. The patch might look something like the attached (untested). So the debate would be whether the current behavior is best, and if not whether this should be configurable. I tend to come down on the side of "yes", and "no". -Barry -------------------- snip snip -------------------- Index: DSN.py =================================================================== RCS file: /cvsroot/mailman/mailman/Mailman/Bouncers/DSN.py,v retrieving revision 2.7 diff -u -r2.7 DSN.py --- DSN.py 2 Dec 2002 14:21:58 -0000 2.7 +++ DSN.py 26 Feb 2003 03:18:45 -0000 @@ -1,17 +1,17 @@ -# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2003 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. -# +# # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software +# along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. """Parse RFC 1894 (i.e. DSN) bounce formats.""" @@ -20,6 +20,8 @@ from email.Utils import parseaddr from cStringIO import StringIO +from Mailman.BouncerAPI import Stop + def check(msg): @@ -39,13 +41,10 @@ # that for other purposes :( # # Also grok out Action so we can do something with that too. - action = msgblock.get('action', '') - # BAW: Should we treat delayed bounces the same? Yes, because if - # the transient problem clears up, they should get unbounced. The - # other problem is what to do about a DSN that has both delayed - # and failed actions in multiple header blocks? We're not - # architected to handle that. ;/ - if action.lower() not in ('failed', 'failure', 'delayed'): + action = msgblock.get('action', '').lower() + if action == 'delayed': + return Stop + if action not in ('failed', 'failure'): # Some non-permanent failure, so ignore this block continue params = []
participants (5)
-
barry＠python.org -
Chuq Von Rospach -
Dan Mick -
M.-A. Lemburg -
Ross Boylan