needs explanation of the interaction between "site list" and bounces

Hello,
(sorry to post here; I've been trying to re-subscribe to mailman-users for two days, w/o luck, and I really need help -- can someone check why I still can't post there?)
I'm looking for a description of the interactions between bounce processing and the site list (mailman@...) under 2.1.11rc1. It seems to me that there is a case when the bounce processor doesn't recognize a bounce, sends it to ??? which in turn generates an error like "you are not allowed to post to the list mailman@..."
thanks
-- Fil

Fil wrote:
I'm looking for a description of the interactions between bounce processing and the site list (mailman@...) under 2.1.11rc1. It seems to me that there is a case when the bounce processor doesn't recognize a bounce, sends it to ??? which in turn generates an error like "you are not allowed to post to the list mailman@..."
Here's the situation. Ordinarily, any unrecognized bounce may be forwarded to the list owner depending on the list's bounce_unrecognized_goes_to_list_owner setting. Nothing special here.
Where the site list comes in is not with unrecognized bounces per se. It is involved with what happens when any message to the list owner bounces.
Any message sent to the list-owner has an envelope sender set to the sitelist-bounces address, so if a message sent to a list-owner bounces, the bounce (assuming a conformant MTA is bouncing) goes to the mailman-bounces@... (or whatever the sitelist name is) address and is handled as a bounce for that list. If it is recognized, it is scored or discarded depending on whether or not the bounced address is a member of the site list.
Now it gets tricky. If the bounce returned to the site list is unrecognized and the sitelist is set to forward unrecognized bounces, the bounce is forwarded. Now that forward may bounce too, so it is sent with a special envelope from mailman-loop@... (or whatever the sitelist name is) so if it bounces, the bounce should be returned to the loop address and every installation should have an alias
# The ultimate loop stopper address mailman-loop: /usr/local/mailman/data/owner-bounces.mbox
to deliver these bounces to a mailbox.
There are certain other complications to avoid the loop that will result if a list owner sets the list-bounces address as the 'owner' address. This is avoided by looking for an X-BeenThere: header for the list in the headers of the 'bounce'. The theory is that no real bounce will have any X-BeenThere: header, so if this does, it came directly from the list to the list-bounces address. In this case, it isn't processed as a bounce, but is sent directly to the sitelist-owner with envelope fron sitelist-loop.
There is no case where an unrecognized bounce should go to the site list itself unless the sitelist is the owner of the sitelist or of the list that sent the bounced message.
There are several comments (and of course the code itself) at the beginning of BounceRunner._process() that give more detail.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark Sapiro wrote:
There are certain other complications to avoid the loop that will result if a list owner sets the list-bounces address as the 'owner' address. This is avoided by looking for an X-BeenThere: header for the list in the headers of the 'bounce'. The theory is that no real bounce will have any X-BeenThere: header, so if this does, it came directly from the list to the list-bounces address. In this case, it isn't processed as a bounce, but is sent directly to the sitelist-owner with envelope fron sitelist-loop.
Ooops. It isn't sent to sitelist-owner. It is sent to the site list.
There is no case where an unrecognized bounce should go to the site list itself unless the sitelist is the owner of the sitelist or of the list that sent the bounced message.
Wrong! If any message comes to bounce processing that is To: the sitelist-bounces address (i.e., a bounce of a list-owner message or a bounce of a message from the site list) or is for a list but has an X-BeenThere header for that list, it will be sent to the site list (not the site list owner) with envelope from sitelist-loop.
Note that this means that normal bounce processing doesn't work for the site list. This shouldn't be a real hardship because members of the site list should all be Mailman admins with good addresses anyway.
The problem that makes this necessary is that when a bounce arrives, we can't really tell in every case where the original, bounced message was sent or even if it is a real DSN.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 | Mark Sapiro wrote: |> |>There are certain other complications to avoid the loop that will |>result if a list owner sets the list-bounces address as the 'owner' |>address. This is avoided by looking for an X-BeenThere: header for the |>list in the headers of the 'bounce'. The theory is that no real bounce |>will have any X-BeenThere: header, so if this does, it came directly |>from the list to the list-bounces address. In this case, it isn't |>processed as a bounce, but is sent directly to the sitelist-owner with |>envelope fron sitelist-loop. | | | Ooops. It isn't sent to sitelist-owner. It is sent to the site list. I've revisited this code in BounceRunner.py and recalled all the misgivings I had the last time I looked at it. Fil and I have been having an off-list exchange about some messages he has seen from a Domino/Lotus server that appears to return a proper DSN except several headers from the bounced message including the List-* headers, Mailman-Version: and X-BeenThere: are included in the message headers of the DSN itself (not just the part of the DSN that contains the original message). This triggered the loop detection logic and mishandled the DSN. I have now changed this code in two ways. I don't rely on X-BeenThere: to detect a potential loop, and I do send these list-owner and looping bounces to the site list owners rather than the site list. A patch to BounceRunner.py incorporating these changes and the list-name logging changes from the "better logging of undiscernable bounces" thread is attached and will be in 2.1.11rc2 next week. - -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFIXcTeVVuXXpU7hpMRAvotAKC9g0mhbVy0esNigcIhjHaG13+/BwCfcDB9 xZG9ZloQPLlE6eB/LhJCYU8= =JBXr -----END PGP SIGNATURE----- === modified file 'Mailman/Queue/BounceRunner.py' --- Mailman/Queue/BounceRunner.py 2007-11-19 20:30:51 +0000 +++ Mailman/Queue/BounceRunner.py 2008-06-22 02:48:44 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2001-2006 by the Free Software Foundation, Inc. +# Copyright (C) 2001-2008 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -178,29 +178,45 @@ # who the message was destined for. That make our job easy. # - the message could have been originally destined for a list owner, # but a list owner address itself bounced. That's bad, and for now - # we'll simply log the problem and attempt to deliver the message to - # the site owner. - # - the list owner could have set listname-bounces as the owner - # address. That's really bad as it results in a loop of ever + # we'll simply attempt to deliver the message to the site list + # owner. + # Note that this means that automated bounce processing doesn't work + # for the site list. Because we can't reliably tell to what address + # a non-VERP'd bounce was originally sent, we have to treat all + # bounces sent to the site list as potential list owner bounces. + # - the list owner could have set list-bounces (or list-admin) as the + # owner address. That's really bad as it results in a loop of ever # growing unrecognized bounce messages. We detect this based on the - # X-BeenThere header and handle it like a list owner bounce. No - # real bounce will have an X-BeenThere header for the list. - bts = [s.strip().lower() for s in msg.get_all('x-beenthere', [])] - if mlist.GetListEmail().lower() in bts: - bt = True - else: - bt = False - # All messages to list-owner@vdom.ain have their envelope sender set - # to site-owner@dom.ain (no virtual domain). Is this a bounce for a - # message to a list owner, coming to the site owner, or an owner - # notice sent directly to the -bounces address? - if msg.get('to', '') == Utils.get_site_email(extra='owner') or bt: + # fact that this message itself will be from the site bounces + # address. We then send this to the site list owner instead. + # Notices to list-owner have their envelope sender and From: set to + # the site-bounces address. Check if this is this a bounce for a + # message to a list owner, coming to site-bounces, or a looping + # message sent directly to the -bounces address. We have to do these + # cases separately, because sending to site-owner will reset the + # envelope sender. + # Is this a site list bounce? + if (mlist.internal_name().lower() == + mm_cfg.MAILMAN_SITE_LIST.lower()): # Send it on to the site owners, but craft the envelope sender to # be the -loop detection address, so if /they/ bounce, we won't # get stuck in a bounce loop. outq.enqueue(msg, msgdata, - recips=[Utils.get_site_email()], - envsender=Utils.get_site_email(extra='loop'), + recips=mlist.owner, + envsender=Utils.get_site_email(extra='loop'), + nodecorate=1, + ) + return + # Is this a possible looping message sent directly to a list-bounces + # address other than the site list? + # Use the From: because message may not have a unix_from. + if msg.get('from') == Utils.get_site_email(extra='bounces'): + # Just send it to the sitelist-owner address. If that bounces + # we'll handle it above. + outq.enqueue(msg, msgdata, + recips=[Utils.get_site_email(extra='owner')], + envsender=Utils.get_site_email(extra='loop'), + nodecorate=1, ) return # List isn't doing bounce processing? @@ -227,8 +243,10 @@ # If that still didn't return us any useful addresses, then send it on # or discard it. if not addrs: - syslog('bounce', 'bounce message w/no discernable addresses: %s', - msg.get('message-id')) + syslog('bounce', + '%s: bounce message w/no discernable addresses: %s', + mlist.internal_name(), + msg.get('message-id', 'n/a')) maybe_forward(mlist, msg) return # BAW: It's possible that there are None's in the list of addresses, @@ -330,8 +348,12 @@ """), subject=_('Uncaught bounce notification'), tomoderators=0) - syslog('bounce', 'forwarding unrecognized, message-id: %s', + syslog('bounce', + '%s: forwarding unrecognized, message-id: %s', + mlist.internal_name(), msg.get('message-id', 'n/a')) else: - syslog('bounce', 'discarding unrecognized, message-id: %s', + syslog('bounce', + '%s: discarding unrecognized, message-id: %s', + mlist.internal_name(), msg.get('message-id', 'n/a'))

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Sapiro wrote: | | I have now changed this code in two ways. I don't rely on X-BeenThere: | to detect a potential loop, and I do send these list-owner and looping | bounces to the site list owners rather than the site list. | | A patch to BounceRunner.py incorporating these changes and the list-name | logging changes from the "better logging of undiscernable bounces" | thread is attached and will be in 2.1.11rc2 next week. And I've made yet another change. It turns out the the BounceRunner.patch2.txt patch would fail to detect the unrecognized bounce loop resulting from a list's owner being set to list-owner@... or list-admin@ ... if the list was in a virtual domain. The attached BounceRunner.patch2.patch.txt corrects the previous patch to fix this issue, and the attached BounceRunner.patch3.txt is a cumulative patch starting with the logging changes. - -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFIXrfTVVuXXpU7hpMRAlQjAKC/6VNvPGDAzJEhDPb9vr988eWidACcDGMl UyaXqiXJUrjmg7+UPSK0BT4= =34xR -----END PGP SIGNATURE----- === modified file 'Mailman/Queue/BounceRunner.py' --- Mailman/Queue/BounceRunner.py 2008-06-22 16:53:02 +0000 +++ Mailman/Queue/BounceRunner.py 2008-06-22 19:22:27 +0000 @@ -209,8 +209,10 @@ return # Is this a possible looping message sent directly to a list-bounces # address other than the site list? - # Use the From: because message may not have a unix_from. - if msg.get('from') == Utils.get_site_email(extra='bounces'): + # Check From: because unix_from might be VERP'd. + # Also, check the From: that Message.OwnerNotification uses. + if (msg.get('from') == + Utils.get_site_email(mlist.host_name, 'bounces')): # Just send it to the sitelist-owner address. If that bounces # we'll handle it above. outq.enqueue(msg, msgdata, === modified file 'Mailman/Queue/BounceRunner.py' --- Mailman/Queue/BounceRunner.py 2007-11-19 20:30:51 +0000 +++ Mailman/Queue/BounceRunner.py 2008-06-22 19:22:27 +0000 @@ -1,4 +1,4 @@ -# Copyright (C) 2001-2006 by the Free Software Foundation, Inc. +# Copyright (C) 2001-2008 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -178,29 +178,47 @@ # who the message was destined for. That make our job easy. # - the message could have been originally destined for a list owner, # but a list owner address itself bounced. That's bad, and for now - # we'll simply log the problem and attempt to deliver the message to - # the site owner. - # - the list owner could have set listname-bounces as the owner - # address. That's really bad as it results in a loop of ever + # we'll simply attempt to deliver the message to the site list + # owner. + # Note that this means that automated bounce processing doesn't work + # for the site list. Because we can't reliably tell to what address + # a non-VERP'd bounce was originally sent, we have to treat all + # bounces sent to the site list as potential list owner bounces. + # - the list owner could have set list-bounces (or list-admin) as the + # owner address. That's really bad as it results in a loop of ever # growing unrecognized bounce messages. We detect this based on the - # X-BeenThere header and handle it like a list owner bounce. No - # real bounce will have an X-BeenThere header for the list. - bts = [s.strip().lower() for s in msg.get_all('x-beenthere', [])] - if mlist.GetListEmail().lower() in bts: - bt = True - else: - bt = False - # All messages to list-owner@vdom.ain have their envelope sender set - # to site-owner@dom.ain (no virtual domain). Is this a bounce for a - # message to a list owner, coming to the site owner, or an owner - # notice sent directly to the -bounces address? - if msg.get('to', '') == Utils.get_site_email(extra='owner') or bt: + # fact that this message itself will be from the site bounces + # address. We then send this to the site list owner instead. + # Notices to list-owner have their envelope sender and From: set to + # the site-bounces address. Check if this is this a bounce for a + # message to a list owner, coming to site-bounces, or a looping + # message sent directly to the -bounces address. We have to do these + # cases separately, because sending to site-owner will reset the + # envelope sender. + # Is this a site list bounce? + if (mlist.internal_name().lower() == + mm_cfg.MAILMAN_SITE_LIST.lower()): # Send it on to the site owners, but craft the envelope sender to # be the -loop detection address, so if /they/ bounce, we won't # get stuck in a bounce loop. outq.enqueue(msg, msgdata, - recips=[Utils.get_site_email()], - envsender=Utils.get_site_email(extra='loop'), + recips=mlist.owner, + envsender=Utils.get_site_email(extra='loop'), + nodecorate=1, + ) + return + # Is this a possible looping message sent directly to a list-bounces + # address other than the site list? + # Check From: because unix_from might be VERP'd. + # Also, check the From: that Message.OwnerNotification uses. + if (msg.get('from') == + Utils.get_site_email(mlist.host_name, 'bounces')): + # Just send it to the sitelist-owner address. If that bounces + # we'll handle it above. + outq.enqueue(msg, msgdata, + recips=[Utils.get_site_email(extra='owner')], + envsender=Utils.get_site_email(extra='loop'), + nodecorate=1, ) return # List isn't doing bounce processing? @@ -227,8 +245,10 @@ # If that still didn't return us any useful addresses, then send it on # or discard it. if not addrs: - syslog('bounce', 'bounce message w/no discernable addresses: %s', - msg.get('message-id')) + syslog('bounce', + '%s: bounce message w/no discernable addresses: %s', + mlist.internal_name(), + msg.get('message-id', 'n/a')) maybe_forward(mlist, msg) return # BAW: It's possible that there are None's in the list of addresses, @@ -330,8 +350,12 @@ """), subject=_('Uncaught bounce notification'), tomoderators=0) - syslog('bounce', 'forwarding unrecognized, message-id: %s', + syslog('bounce', + '%s: forwarding unrecognized, message-id: %s', + mlist.internal_name(), msg.get('message-id', 'n/a')) else: - syslog('bounce', 'discarding unrecognized, message-id: %s', + syslog('bounce', + '%s: discarding unrecognized, message-id: %s', + mlist.internal_name(), msg.get('message-id', 'n/a'))
participants (2)
-
Fil
-
Mark Sapiro