Re: [Mailman-Developers] Missed Bounce Message

On Mon, 29 Nov 1999 22:08:10 +0100 Ricardo Kustner <ricardo@miss-janet.com> wrote:
tell me about it...
As there is no standard format for bounce messages, and even if there were one we'd still be suffering with non-standard ones due to the great many legacy MTAs as well as the many sites with locally customised bounce messages, another approach is needed.
The most commonly recommended solution is VERP. The problem with VERP is that it places a very high load on the local MTA as well as any distribution management you may have set up (eg domain routing). The approach I took back when I was writing my own mailing list server was a middle ground with the intent of avoiding most of VERP's bandwidth concerns while retaining many of its advantages:
The list server emitted messages in bundles of 100 RCPT TOs in the normal manner except for every Nth message.
For every N'th message (and this encluded digests), that message was sent individually to every subscriber with a custom header field added.
The custom header field was ala:
x-bounce-detect: ListMgr <listname> <subscriber_address> <count> <score>
Where 'count' was just a number that incremented every time N rolled around. It was used to detect how old this bounce was and if there had been any other "bounce-sensitive" messages emitted since that one was received. The 'score' value was just a tracking value indicating how close that subscriber was to being unsubscribed.
The intent was that the custom header could be found in the copy of the original email encluded in the bounce message, and could be easily analysed to find the name of the list, the original subscriber's address (in case of the message being forwarded), etc. I then just scanned bounce messages for the header, and kept track of how often and rapidly each subscriber bounced messages, and if the rate exceeded a certain value, unsubscribing that member.
-- J C Lawrence Home: claw@kanga.nu ----------(*) Other: coder@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

"claw" == <claw@kanga.nu> writes:
claw> As there is no standard format for bounce messages, and even
claw> if there were one we'd still be suffering with non-standard
claw> ones due to the great many legacy MTAs as well as the many
claw> sites with locally customised bounce messages, another
claw> approach is needed.
Well, there is DSN, but you're essentially right since most MTA don't seem to do DSN at all. I think there's even some controversy over the complexity of DSN bounce formats, so you're essentially left with a plethora of formats that you have to parse.
claw> The most commonly recommended solution is VERP. The problem
claw> with VERP is that it places a very high load on the local
claw> MTA as well as any distribution management you may have set
claw> up (eg domain routing). The approach I took back when I was
claw> writing my own mailing list server was a middle ground with
claw> the intent of avoiding most of VERP's bandwidth concerns
claw> while retaining many of its advantages:
For a while now I've been thinking that we could do this kind of VERP-like custom messages for the password reminders. Those we already have to craft specially for each recipient, so why not include an easily detected header, then catch those bounces and scan them?
-Barry

Note that some mail servers refuse to send the message body back (grrr). If you do it as the envelope you're guaranteed to be able to recognize it, whether or not the body bounces..
[I'm also highly in favor of the at-password-time bounce checking - keeps load light-ish for most list traffic, and just the occasional reminder/probe]

On Tue, Nov 30, 1999 at 01:29:57PM -0800, jfesler@gigo.com wrote:
Yeah, that sounds like a good idea, but are you going to kick them off after one bounce, or wait several months before kicking them off? Or would a failure of that message cause probes to begin being sent, or the address to be marked as "suspect" and use VERPs for delivery?
I'm imagining something like a binary search. You *CAN* use a header or VERP while still sending batches, you just have to realize that a VERP is associated with a particular SET of addresses. You mark these as suspect, and future deliveries to these addresses aren't batched. The back-off time on these addresses being marked as suspect can be relatively low, but probably should be by number of messages sent, not by hours (for low volume lists). Maybe 3 messages or less.
So, if you have three addresses bouncing from among 10k addresses, and your chunk size is 20, you only send out about an extra 150 messages to determin exactly where the bounce is.
This seems more efficient than sending probes regularly (particularly for huge lists), and more pro-active than waiting till the end of the month.
I'd be interested in working on such a thing. My question though is, what's the preferred way to do this? A header line (which may be truncated by some MTAs), a unique envelope sender (which requires some support from the *LOCAL* MTA, and qmail and sendmail do it differently)?
Sean
Her eyes were like two brown circles with big black dots in the center. Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.

On Tue, 30 Nov 1999 13:29:57 -0800 (PST) jfesler <jfesler@gigo.com> wrote:
Note that some mail servers refuse to send the message body back (grrr).
I've found that the ones that don't at least send the headers are few and far between (tho they do exist).
If you do it as the envelope you're guaranteed to be able to recognize it, whether or not the body bounces..
True.
As I mentioned to Barry, I opted for once every N messages where N was configurable on a per-list basis. This allowed some lists to be set trigger happy, and others lackadasical -- as befitted their character and daily traffic loads.
Hurm.
My headers were:
X-SubscriberData: subscr_address on <listname> #<lists's_bounce_count>"
The algorithm was:
There were three configurable values per list: How often to issue bounce messages: Every N days or M messages, whichever came the sooner (no mail days genereated no bounce test messages), and a sensitivity value.
A bounced message was worth Q points, where Q was determined as follows:
To be removed from the list a member had to accrue a sufficiently high bounce score. The more rapidly he bounced messages the more rapidly his score grew. Gaps in bounces (is some bounce some don't, and the score grows more slowly.
Loosely, a mamber had to bounce every bounce_message for <sensitivity> days with the bounce messages making it back to the list server within 24 hours to be unsubscribed.
This was done by calculating a bounce_score as follows:
If the bounce message has the same bounce_count as the last
bounce-sensitive message issued (plus or minus one), then <sensitivity> is added to the score.
If the bounc message is from an older bounce-test, then
<sensitivity> - <age> - 1 is added to the score.
If the bounce message is older than <sensitivity> days, then the
score is set to zero.
-- J C Lawrence Home: claw@kanga.nu ----------(*) Other: coder@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--

"claw" == <claw@kanga.nu> writes:
claw> As there is no standard format for bounce messages, and even
claw> if there were one we'd still be suffering with non-standard
claw> ones due to the great many legacy MTAs as well as the many
claw> sites with locally customised bounce messages, another
claw> approach is needed.
Well, there is DSN, but you're essentially right since most MTA don't seem to do DSN at all. I think there's even some controversy over the complexity of DSN bounce formats, so you're essentially left with a plethora of formats that you have to parse.
claw> The most commonly recommended solution is VERP. The problem
claw> with VERP is that it places a very high load on the local
claw> MTA as well as any distribution management you may have set
claw> up (eg domain routing). The approach I took back when I was
claw> writing my own mailing list server was a middle ground with
claw> the intent of avoiding most of VERP's bandwidth concerns
claw> while retaining many of its advantages:
For a while now I've been thinking that we could do this kind of VERP-like custom messages for the password reminders. Those we already have to craft specially for each recipient, so why not include an easily detected header, then catch those bounces and scan them?
-Barry

Note that some mail servers refuse to send the message body back (grrr). If you do it as the envelope you're guaranteed to be able to recognize it, whether or not the body bounces..
[I'm also highly in favor of the at-password-time bounce checking - keeps load light-ish for most list traffic, and just the occasional reminder/probe]

On Tue, Nov 30, 1999 at 01:29:57PM -0800, jfesler@gigo.com wrote:
Yeah, that sounds like a good idea, but are you going to kick them off after one bounce, or wait several months before kicking them off? Or would a failure of that message cause probes to begin being sent, or the address to be marked as "suspect" and use VERPs for delivery?
I'm imagining something like a binary search. You *CAN* use a header or VERP while still sending batches, you just have to realize that a VERP is associated with a particular SET of addresses. You mark these as suspect, and future deliveries to these addresses aren't batched. The back-off time on these addresses being marked as suspect can be relatively low, but probably should be by number of messages sent, not by hours (for low volume lists). Maybe 3 messages or less.
So, if you have three addresses bouncing from among 10k addresses, and your chunk size is 20, you only send out about an extra 150 messages to determin exactly where the bounce is.
This seems more efficient than sending probes regularly (particularly for huge lists), and more pro-active than waiting till the end of the month.
I'd be interested in working on such a thing. My question though is, what's the preferred way to do this? A header line (which may be truncated by some MTAs), a unique envelope sender (which requires some support from the *LOCAL* MTA, and qmail and sendmail do it differently)?
Sean
Her eyes were like two brown circles with big black dots in the center. Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com> URL: <http://www.tummy.com/xvscan> HP-UX/Linux/FreeBSD/BSDOS scanning software.

On Tue, 30 Nov 1999 13:29:57 -0800 (PST) jfesler <jfesler@gigo.com> wrote:
Note that some mail servers refuse to send the message body back (grrr).
I've found that the ones that don't at least send the headers are few and far between (tho they do exist).
If you do it as the envelope you're guaranteed to be able to recognize it, whether or not the body bounces..
True.
As I mentioned to Barry, I opted for once every N messages where N was configurable on a per-list basis. This allowed some lists to be set trigger happy, and others lackadasical -- as befitted their character and daily traffic loads.
Hurm.
My headers were:
X-SubscriberData: subscr_address on <listname> #<lists's_bounce_count>"
The algorithm was:
There were three configurable values per list: How often to issue bounce messages: Every N days or M messages, whichever came the sooner (no mail days genereated no bounce test messages), and a sensitivity value.
A bounced message was worth Q points, where Q was determined as follows:
To be removed from the list a member had to accrue a sufficiently high bounce score. The more rapidly he bounced messages the more rapidly his score grew. Gaps in bounces (is some bounce some don't, and the score grows more slowly.
Loosely, a mamber had to bounce every bounce_message for <sensitivity> days with the bounce messages making it back to the list server within 24 hours to be unsubscribed.
This was done by calculating a bounce_score as follows:
If the bounce message has the same bounce_count as the last
bounce-sensitive message issued (plus or minus one), then <sensitivity> is added to the score.
If the bounc message is from an older bounce-test, then
<sensitivity> - <age> - 1 is added to the score.
If the bounce message is older than <sensitivity> days, then the
score is set to zero.
-- J C Lawrence Home: claw@kanga.nu ----------(*) Other: coder@kanga.nu --=| A man is as sane as he is dangerous to his environment |=--
participants (4)
-
Barry A. Warsaw
-
claw@kanga.nu
-
jfesler@gigo.com
-
Sean Reifschneider