Doubt regarding definition of "temporary and permanent failures"
Are "temporary and permanent failures" mentioned in "bounce.py" in "mailman/src/mailman/runners/" the same as "soft and hard bounces" respectively?
On 3/11/19 1:45 PM, Aaryan Bhagat wrote:
Are "temporary and permanent failures" mentioned in "bounce.py" in "mailman/src/mailman/runners/" the same as "soft and hard bounces" respectively?
No. A temporary failure is a 4xx (retryable) status and will cause mailman to retry periodically for a configured time (defaults I think are 15 minutes and 5 days). If the message is still not delivered after the 5 days or whatever time, it is considered a bounce at that point.
A permanent failure is a 5xx (failure) status and is recorded as a bounce. All 5xx failures or failure DSNs are considered bounces. "Soft" reasons like full mailbox or message looks like spam are not distinguished from "hard" reasons like no such user.
Whether or not a failure is considered retryable is up to the receiving MTA.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Is this in the mailman3 doc? Like for what failure should we retry or discard the mail etc.
On Tue, 12 Mar 2019, 04:02 Mark Sapiro, <mark@msapiro.net> wrote:
On 3/11/19 1:45 PM, Aaryan Bhagat wrote:
Are "temporary and permanent failures" mentioned in "bounce.py" in "mailman/src/mailman/runners/" the same as "soft and hard bounces" respectively?
No. A temporary failure is a 4xx (retryable) status and will cause mailman to retry periodically for a configured time (defaults I think are 15 minutes and 5 days). If the message is still not delivered after the 5 days or whatever time, it is considered a bounce at that point.
A permanent failure is a 5xx (failure) status and is recorded as a bounce. All 5xx failures or failure DSNs are considered bounces. "Soft" reasons like full mailbox or message looks like spam are not distinguished from "hard" reasons like no such user.
Whether or not a failure is considered retryable is up to the receiving MTA.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-leave@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3
Security Policy: https://wiki.list.org/x/QIA9
On 3/11/19 8:32 PM, aaryan bhagat wrote:
Is this in the mailman3 doc?
There is some doc at <https://mailman.readthedocs.io/en/latest/src/mailman/model/docs/bounce.html>.
Like for what failure should we retry or discard the mail etc.
There are two kinds of bounces. There are failures in SMTP delivery from Mailman to the MTA. These are all handled by the outgoing runner. 5xx permanent failures are registered as bounces and 4xx temporary failures are retried.
The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce <https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
There is some doc at <https://mailman.readthedocs.io/en/latest/src/mailman/model/docs/bounce.html>
I have have gone through this earlier, here there is only mention about the registration of bounce events in a queue for processing.
No mention of the type of errors discussed above.
If the message is still not delivered after the 5 days or whatever time, it is considered a bounce at that point.
The point is I have to design this workflow of what exactly will happen and these inputs I need in a concrete form for finalizing. Although for this case we can decide later but for some others we may need now.
On 3/12/19 2:21 AM, Aaryan Bhagat wrote:
There is some doc at <https://mailman.readthedocs.io/en/latest/src/mailman/model/docs/bounce.html>
I have have gone through this earlier, here there is only mention about the registration of bounce events in a queue for processing.
No mention of the type of errors discussed above.
Right. This is not well documented. Part of this project is to improve that.
If the message is still not delivered after the 5 days or whatever time, it is considered a bounce at that point.
The point is I have to design this workflow of what exactly will happen and these inputs I need in a concrete form for finalizing. Although for this case we can decide later but for some others we may need now.
No. You don't need to do that. It is already done. See my next reply.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce <https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
In the end the bounce runner gets all the messages for a particular list and ( irrespective of who send them DSN or failure in SMTP). So I think dealing with them is our main objective.
On 3/12/19 3:57 AM, Aaryan Bhagat wrote:
The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce <https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
In the end the bounce runner gets all the messages for a particular list and ( irrespective of who send them DSN or failure in SMTP). So I think dealing with them is our main objective.
The receipt and handling of delivery failures and bounces is already
well handled in Mailman 3 and results in a bounce event being stored.
It is what happens after that that is missing.
In Mailman 2.1 this is simpler in a way because addresses are local to a list. I.e., there is no connection between an address subscribed to one list and the same address subscribed to another list so the whole process is compartmentalized by list and controlled by list settings. These settings are:
The maximum member bounce score before the member's subscription is disabled. This value can be a floating point number. (Details for bounce_score_threshold)
The number of days after which a member's bounce information is discarded, if no new bounces have been received in the interim. This value must be an integer. (Edit bounce_info_stale_after)
How many Your Membership Is Disabled warnings a disabled member should get before their address is removed from the mailing list. Set to 0 to immediately remove an address from the list once their bounce score exceeds the threshold. This value must be an integer. (Edit bounce_you_are_disabled_warnings)
The number of days between sending the Your Membership Is Disabled warnings. This value must be an integer. (Edit bounce_you_are_disabled_warnings_interval)
These settings are per list both to give list admins control and because optimum values depend on list traffic patterns.
Mailman 3 currently does nothing after storing a 'bounce event'. That's the issue. It's more complex with Mailman 3 because an address is global, but somehow, it needs to be determined that we are receiving bounces for a particular address and at some point delivery to that address needs to be disabled (or a probe sent to the address and delivery disabled if the probe bounces) and then periodically, notices need to be sent to the address, and after some time if the user hasn't responded and re-enabled delivery, perhaps the address should be removed from the list, although this latter point is not so important in MM 3.
The additional complexity is because an address may be subscribed to more than one list and optimum values for settings such as bounce_score_threshold and bounce_info_stale_after depend on list traffic patterns. Compromises will need to be made.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks for the guidance mark!
Also
The additional complexity is because an address may be subscribed to more than one list and optimum values for settings such as bounce_score_threshold and bounce_info_stale_after depend on list traffic patterns. Compromises will need to be made.
Cannot we have separate variables managed by specific list owners/moderators for this type of user.
On 3/12/19 10:01 AM, Aaryan Bhagat wrote:
Thanks for the guidance mark!
Also
The additional complexity is because an address may be subscribed to more than one list and optimum values for settings such as bounce_score_threshold and bounce_info_stale_after depend on list traffic patterns. Compromises will need to be made.
Cannot we have separate variables managed by specific list owners/moderators for this type of user.
We can but how does it all fit together and make sense? In a MM 3 installation we only have one address record for an address regardless of how many lists it is a member of, and if that address is bouncing on one list, presumably it will bounce on all lists. So do we want to keep track of bounces per address per list and disable delivery per list, or does it make more sense to keep track of bounces per address and disable all delivery to the address?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I get your point mark making a joint record is important like if an address has 1 bounce each from n lists then the number should be n and we should delete it instead saying the number is 1 for each and we should not take any action.
On Tue, 12 Mar 2019, 23:00 Mark Sapiro, <mark@msapiro.net> wrote:
Thanks for the guidance mark!
Also
The additional complexity is because an address may be subscribed to more than one list and optimum values for settings such as bounce_score_threshold and bounce_info_stale_after depend on list traffic
On 3/12/19 10:01 AM, Aaryan Bhagat wrote: patterns. Compromises will need to be made.
Cannot we have separate variables managed by specific list
owners/moderators for this type of user.
We can but how does it all fit together and make sense? In a MM 3 installation we only have one address record for an address regardless of how many lists it is a member of, and if that address is bouncing on one list, presumably it will bounce on all lists. So do we want to keep track of bounces per address per list and disable delivery per list, or does it make more sense to keep track of bounces per address and disable all delivery to the address?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-leave@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3
Security Policy: https://wiki.list.org/x/QIA9
On 3/12/19 10:37 AM, aaryan bhagat wrote:
I get your point mark making a joint record is important like if an address has 1 bounce each from n lists then the number should be n and we should delete it instead saying the number is 1 for each and we should not take any action.
Maybe yes, maybe no. What if all n bounces were of a single post cross-posted to n lists?
One thing we do in MM 2.1, albeit on a per-list basis, is only score the first bounce for an address on any given day. This is to address the situation where a transient mis-configuration might cause several bounces in a short time but then be fixed.
These are all complex questions with no easy "one size fits all" answers.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Aaryan Bhagat writes:
The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce <https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
In the end the bounce runner gets all the messages for a particular list and ( irrespective of who send them DSN or failure in SMTP). So I think dealing with them is our main objective.
In some sense, yes. Mark's point is that receipt of SMTP failures and DSNs is already recorded as a "bounce event". The classification of the various messages etc into bounce events we use has been working for literally decades; there is no need to revist it now. If you want to understand these well, you should read the RFCs on the mail system, such as RFC 5598 on the Internet Mail Architecture, RFC 5321 on SMTP, RFC 5322 on the Internet Message Format, and RFC 3464 on Delivery Status Notifications (which even today controls only a large fraction of DSNs, so we still receive non-standard DSNs). But all that study is *not* needed to work on Mailman; most of it is already encapsulated in Mailman 3 code.
However, with Mailman 3 the interpretation of a bounce event may need to be changed because we now associate addresses and subscriptions with users. A bounce event may be thought of as a (from, to, timestamp) tuple, where "from" is the list, "to" is the subscribed address, and "timestamp" is the datetime of the event. (We only record how many have occurred, and how long since the last one.) In Mailman 2, only one subscriber-list pair is affected by the bounce count. This *may* make sense for Mailman 3 too, because (for example) one list has a lot of commercial traffic that looks spammy and generates "administrative refusal" bounces (perhaps because many users at one domain are receiving it) even though the user really wants it, while another list from the same site may be low-traffic and not generate bounces. Then the traditional Mailman 2 method of keeping bounce counts per subscriber per list makes sense.
On the other hand, it might be that bounces for subscriber addresses are highly correlated across lists at the same site (eg, if mostly they are due to storage limitations). Then we want to record bounce counts per address, summing over lists. This is plausible, and some site administrators who are very sensitive to bounces (ie, want to disable or unsubscribe failing addresses quickly) might want this as an option, even if most sites want to use the per subscriber per list style. However, this could also be implemented by keeping per- subscription counts, and adding them up, at the cost of multiple hits to the database to find all subscriptions, then get all the bounce counts.
We might even want to record bounce counts per user (ie, adding up counts for all addresses for that user), on the assumption that all addresses forward to the same "real" address, and most bounces are due to problems at that address. I think this is a bad idea, as in some cases automatic forwards are copies, and the user can read them at the subscribed address even though they normally read them at the "real" address. Such users would be really angry if a non-bouncing address got disabled! However, it gives you an idea of the scope of the kinds of issues we want to be considering now.
Steve
-- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
Much appreciated Steve! I have a idea of what is there to work on and will post new doubt on threads soon!.
In "bounce.py" in "mailman/src/mailman/runners"
There is a comment in line 75 of this file states--------- The addresses will come back from flufl.bounce as bytes/8-bit strings, but we must store them as unicodes in the database. Assume utf-8 encoding, but be cautious.
Reply by Mark Sapiro
There are two kinds of bounces. There are failures in SMTP delivery from Mailman to the MTA. These are all handled by the outgoing runner. 5xx permanent failures are registered as bounces and 4xx temporary failures are retried. The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce <https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
Since there are two types of bounces. What is the return format of the first type ( failures in SMTP )? Do we need some encoding to their response also?
On 3/12/19 4:17 AM, Aaryan Bhagat wrote:
In "bounce.py" in "mailman/src/mailman/runners"
There is a comment in line 75 of this file states--------- The addresses will come back from flufl.bounce as bytes/8-bit strings, but we must store them as unicodes in the database. Assume utf-8 encoding, but be cautious.
Reply by Mark Sapiro
There are two kinds of bounces. There are failures in SMTP delivery from Mailman to the MTA. These are all handled by the outgoing runner. 5xx permanent failures are registered as bounces and 4xx temporary failures are retried. The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce <https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
Since there are two types of bounces. What is the return format of the first type ( failures in SMTP )? Do we need some encoding to their response also?
You are focusing on the wrong thing. All the existing code is doing the right thing. It is the part that isn't implemented you that you need to be concerned with. I.e., once a bounce event is recorded, what do we do next.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Of course I will focus on what you explained me earlier. I just wanted to get a detailed understanding og what it is doing already. Thanks for help!
On Tue, 12 Mar 2019, 22:47 Mark Sapiro, <mark@msapiro.net> wrote:
In "bounce.py" in "mailman/src/mailman/runners"
There is a comment in line 75 of this file states--------- The addresses will come back from flufl.bounce as bytes/8-bit strings, but we must store them as unicodes in the database. Assume utf-8 encoding, but be cautious.
Reply by Mark Sapiro
There are two kinds of bounces. There are failures in SMTP delivery from Mailman to the MTA. These are all handled by the outgoing runner. 5xx
On 3/12/19 4:17 AM, Aaryan Bhagat wrote: permanent failures are registered as bounces and 4xx temporary failures are retried. The other kind are DSNs of some type returned to a LIST-bounces address. These are processed by flufl.bounce < https://gitlab.com/warsaw/flufl.bounce/> which determines whether the failure is temporary or permanent and reports that to the bounce runner.
Since there are two types of bounces. What is the return format of the
first type ( failures in SMTP )?
Do we need some encoding to their response also?
You are focusing on the wrong thing. All the existing code is doing the right thing. It is the part that isn't implemented you that you need to be concerned with. I.e., once a bounce event is recorded, what do we do next.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-leave@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3
Security Policy: https://wiki.list.org/x/QIA9
On 3/12/19 10:18 AM, aaryan bhagat wrote:
Of course I will focus on what you explained me earlier. I just wanted to get a detailed understanding og what it is doing already. Thanks for help!
To answer the question ...
> Since there are two types of bounces. What is the return format of the first type ( failures in SMTP )? > Do we need some encoding to their response also?
runners/bounce.py never sees the failures in smtp delivery. They are handled in runners/outgoing.py and are not processed by flufl.bounce at all. In that case, runners/outgoing.py gets the addresses of temporary and permanent failures in the error response when mta/deliver.py raises SomeRecipientsFailed
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (4)
-
aaryan bhagat -
Aaryan Bhagat
-
Mark Sapiro -
Stephen J. Turnbull