
One of the last major subsystems that I need to get working in Mailman 3 is bounce processing. This is different than bounce detection, which has been successfully ported from Mailman 2, but doesn't differ in any significant way. The question I am thinking about now is what to do with a bounce once we detect one.
There are a couple of interesting things in MM3 that makes it different from MM2. In MM3, users and addresses are global to the system, while membership is specific to a mailing list. This means if we register a bounce on an address, we can have that score affect the address's subscription to all relevant mailing list. We can also do things like automatically roll over to another registered and validated email address for that user, if there is one, or at least send notifications to the other address.
There's also the question about how all the bounce scores are managed, and the knobs you as a list administrator can tweak to control how and when things happen based on the score. Reporting and logging are also part of the plan.
Because MM3 uses a relational database underneath the hood, my plan is to have a single table that only appends new bounce events. That way, Mailman will have a permanent record of every bounce that occurred. Exactly what information we can or should put in that table is up for discussion. I do plan also to keep all bounce messages in MM3's "message store" so that postmortem debugging is easier.
Because I'm just starting to think about all this, I wanted to throw this out to the list to get your feedback on things you'd like to see. What is it about MM2's bounce processing that you like? What don't you like? What MM2 bounce features can you do without? What would you like to see added?
Any and all feedback is welcome. -Barry

- Barry Warsaw <barry@list.org>:
Here's an "I think while I type" section... ;)
Should we apply a global action to a local problem i.e. if an address in one list bounces, suspend that address in all lists? Sounds like a good feature to keep the mail system from wasting ressources, but is this a good service for the recipient?
If a message cannot be delivered to a specific address is undeliverable it usually is because
- the receiving mail system is down -> global action, because no list will be able to deliver a message
- the recipient does not exist anymore -> global action, because no list will be able to deliver a message
- the recipient never existed -> paradox: User never was able to verify list membership. Maybe she was, with another list and we just took the address for granted. We should always confirm an address for deliverablity reasons!
- the recipients mailbox is over quota -> global action, because no list will be able to deliver a message
- the envelope sender is being rejected -> problem! We can't tell if the specific envelope sender or the whole envelope sender domain was rejected. Action: Local. In dubio pro reo. In this case I'd leave it to all lists to learn that the particular recipient is undeliverable.
We could apply a score and disable sending in all lists if e.g. three lists detected the recipient address bounces. But what's the consequence? Once we disable one recipient, should we apply that to all recipients in that recipients domain? Looks like a great admin service, but also like a great opportunity to shoot oneself into the foot automatically. It should be a feature that must be called manually.
Send priority by reputation. Reputation deducted from deliverabilty or should I better say receivability? We could create a domains or even a hosts receivabilty record. Those with the worst record are the once we send to last, because they likely will clutter our MSAs outgoing mailqueue.
On a sidenote: I'd welcome an abuse database to blacklist addresses in a global range.
And I like the idea that each MM instance should offer these data as service to other installations. Site-wide. Cross-Site.
I am not acquainted with MM2 bounce features.
p@rick
-- state of mind Digitale Kommunikation
Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666
Amtsgericht München Partnerschaftsregister PR 563

On Fri, Jan 07, 2011 at 07:30:51PM -0500, Barry Warsaw wrote:
This makes me feel slightly uncomfortable, particularly for large (multi-site, multi-client) installations -- Hosting Co, &c.
Hum. If the auto-rollover knew "oh, it's a different MX" there might be a point. However, just trying adam@amyl.org.uk, instead of adam-mailman@amyl.org.uk, or adam+tarpit@amyl.org.uk, would not be very useful, I'd imagine. Maybe an option to specify "this is my recovery address, send bounce-notifications here, please" might be useful? (for end users). It would obviously need to spell out, quite clearly for which address it releated to, as finding an envelope-to: header seems to be tricky for users.
I'm perhaps a little cavalier in my approach; I generally let Mailman handle the bounces, so I can do something useful. About the most I delve, when I don't need to investigate "why aren't I getting mail" is a monthly report of numbers of subscribers, changes to that figure from previous month, and "reasons" why people left, pulled from subscribe.log, at the moment.
What may be useful is to supplement this with a pertinent dates table, too, something like start-date/end-date/few-words-on-problem, either controlled by Chief Goncho (aka site-admins), or maybe with something for listadmins; the case I'm thinking of may be to show that, say the LINX have had problems for a couple of months, "MTA tweak for redelivery attempts to yahoo.com made on 2010-02-04"; these would be added to a gnuplot/graph in a separate color, in my vision (maybe I've used google analytics too long, but clicking on the event for more info would be grand). Perhaps that's function creep, though.
In which case, there should definately be an option for "keep bounce messages in store for N months", and perhaps make list-specific ones available to list-admins.
It does most of the work for me; I set the global parameters, and generally, just leave Mailman to do everything for me. I might sometimes see the "been removed from the list due to bounce" mails; if those were on a grid-thing somewhere in the admin pages, I don't think I'd need/want the mails.
-- "You know it cannot have been a good night when you get into a fight with Spider-Man and two cross-dressing men" -- Mark Davies (defence lawyer, regarding 'Cage fighters picked on because they were dressed as women for a stag night')

- Barry Warsaw <barry@list.org>:
Here's an "I think while I type" section... ;)
Should we apply a global action to a local problem i.e. if an address in one list bounces, suspend that address in all lists? Sounds like a good feature to keep the mail system from wasting ressources, but is this a good service for the recipient?
If a message cannot be delivered to a specific address is undeliverable it usually is because
- the receiving mail system is down -> global action, because no list will be able to deliver a message
- the recipient does not exist anymore -> global action, because no list will be able to deliver a message
- the recipient never existed -> paradox: User never was able to verify list membership. Maybe she was, with another list and we just took the address for granted. We should always confirm an address for deliverablity reasons!
- the recipients mailbox is over quota -> global action, because no list will be able to deliver a message
- the envelope sender is being rejected -> problem! We can't tell if the specific envelope sender or the whole envelope sender domain was rejected. Action: Local. In dubio pro reo. In this case I'd leave it to all lists to learn that the particular recipient is undeliverable.
We could apply a score and disable sending in all lists if e.g. three lists detected the recipient address bounces. But what's the consequence? Once we disable one recipient, should we apply that to all recipients in that recipients domain? Looks like a great admin service, but also like a great opportunity to shoot oneself into the foot automatically. It should be a feature that must be called manually.
Send priority by reputation. Reputation deducted from deliverabilty or should I better say receivability? We could create a domains or even a hosts receivabilty record. Those with the worst record are the once we send to last, because they likely will clutter our MSAs outgoing mailqueue.
On a sidenote: I'd welcome an abuse database to blacklist addresses in a global range.
And I like the idea that each MM instance should offer these data as service to other installations. Site-wide. Cross-Site.
I am not acquainted with MM2 bounce features.
p@rick
-- state of mind Digitale Kommunikation
Franziskanerstraße 15 Telefon +49 89 3090 4664 81669 München Telefax +49 89 3090 4666
Amtsgericht München Partnerschaftsregister PR 563

On Fri, Jan 07, 2011 at 07:30:51PM -0500, Barry Warsaw wrote:
This makes me feel slightly uncomfortable, particularly for large (multi-site, multi-client) installations -- Hosting Co, &c.
Hum. If the auto-rollover knew "oh, it's a different MX" there might be a point. However, just trying adam@amyl.org.uk, instead of adam-mailman@amyl.org.uk, or adam+tarpit@amyl.org.uk, would not be very useful, I'd imagine. Maybe an option to specify "this is my recovery address, send bounce-notifications here, please" might be useful? (for end users). It would obviously need to spell out, quite clearly for which address it releated to, as finding an envelope-to: header seems to be tricky for users.
I'm perhaps a little cavalier in my approach; I generally let Mailman handle the bounces, so I can do something useful. About the most I delve, when I don't need to investigate "why aren't I getting mail" is a monthly report of numbers of subscribers, changes to that figure from previous month, and "reasons" why people left, pulled from subscribe.log, at the moment.
What may be useful is to supplement this with a pertinent dates table, too, something like start-date/end-date/few-words-on-problem, either controlled by Chief Goncho (aka site-admins), or maybe with something for listadmins; the case I'm thinking of may be to show that, say the LINX have had problems for a couple of months, "MTA tweak for redelivery attempts to yahoo.com made on 2010-02-04"; these would be added to a gnuplot/graph in a separate color, in my vision (maybe I've used google analytics too long, but clicking on the event for more info would be grand). Perhaps that's function creep, though.
In which case, there should definately be an option for "keep bounce messages in store for N months", and perhaps make list-specific ones available to list-admins.
It does most of the work for me; I set the global parameters, and generally, just leave Mailman to do everything for me. I might sometimes see the "been removed from the list due to bounce" mails; if those were on a grid-thing somewhere in the admin pages, I don't think I'd need/want the mails.
-- "You know it cannot have been a good night when you get into a fight with Spider-Man and two cross-dressing men" -- Mark Davies (defence lawyer, regarding 'Cage fighters picked on because they were dressed as women for a stag night')
participants (4)
-
Adam McGreggor
-
Barry Warsaw
-
Patrick Ben Koetter
-
Stephen J. Turnbull