Bounce removal parameters default values
I find I am being removed from mailman mailing lists left and right. I believe the default values for the bounce removal should be reconsidered. It's possible that you haven't had many users in my situation and so haven't really had a chance to tune these parameters on the low end yet. But they clearly aren't working for me at a few sites.
My particular situation is that my site has seen fit to filter viruses by refusing delivery. This causes a bounce from the remote MTA every time someone sends me an Outlook virus. Why my site administrators felt this was necessary is a question for another day, it's not like I use Outlook or like my spam filters wouldn't have thrown these messages away anyways, but whatever.
The net result is that some small fraction of messages to me bounce and list management software notices this. The only reason I became aware of the problem was because ezmlm also does this type of processing but it sends a warning message before removing users. It only removes you if the warning message itself bounces. In fact it sends two such warning messages and only removes the user if *both* bounce. This provides the user with a chance to react to the first message and fix the problem -- if they ever see the message.
I could beg for a similar feature in mailman, but I'm not sure it's necessary. But I am sure it's necessary to tune the bounce processing parameters. The relatively few bounces I'm generating shouldn't be causing me to get removed when all the real messages are being delivered fine.
It seems the legitimate messages that are correctly delivered should reset the count of bounces to 0. Reading the source it seems it has to see DEFAULT_MAX_POSTS_BETWEEN_BOUNCES such legitimate posts between messages. I'm fairly convinced this parameter should always be 0. If any successful delivery occurs the user should never be removed due to bounces.
What I don't understand is how DEFAULT_MAX_POSTS_BETWEEN_BOUNCES relates to the parameters I see in the admin. None of the parameters in the admin corresponds to this. How is it calculated?
-- greg
The problem I described in January is still happening. I find the current bounce processing of mailman to be inadequate. Something more like the bounce processing of ezmlm is needed.
I should not be removed from a mailing list purely on the basis of bounces of uncontrolled messages. The messages that bounced could have been spam or outlook worms or whatever.
Before removing a subscriber mailman should send a message with known content testing the address. Only if such a message bounces should a user be dropped.
As it is I'm being removed from mailing lists whenever a new Outlook worm appears and sends multiple messages in a row. Or a new spammer discovers a list I'm on and sends multiple messages in a row to the list.
It's especially bad on low-volume lists where it's quite possible for spam or Outlook worm messages to be the only messages for days.
Greg Stark <gsstark@MIT.EDU> writes:
I find I am being removed from mailman mailing lists left and right. I believe the default values for the bounce removal should be reconsidered. It's possible that you haven't had many users in my situation and so haven't really had a chance to tune these parameters on the low end yet. But they clearly aren't working for me at a few sites.
My particular situation is that my site has seen fit to filter viruses by refusing delivery. This causes a bounce from the remote MTA every time someone sends me an Outlook virus. Why my site administrators felt this was necessary is a question for another day, it's not like I use Outlook or like my spam filters wouldn't have thrown these messages away anyways, but whatever.
The net result is that some small fraction of messages to me bounce and list management software notices this. The only reason I became aware of the problem was because ezmlm also does this type of processing but it sends a warning message before removing users. It only removes you if the warning message itself bounces. In fact it sends two such warning messages and only removes the user if *both* bounce. This provides the user with a chance to react to the first message and fix the problem -- if they ever see the message.
I could beg for a similar feature in mailman, but I'm not sure it's necessary. But I am sure it's necessary to tune the bounce processing parameters. The relatively few bounces I'm generating shouldn't be causing me to get removed when all the real messages are being delivered fine.
It seems the legitimate messages that are correctly delivered should reset the count of bounces to 0. Reading the source it seems it has to see DEFAULT_MAX_POSTS_BETWEEN_BOUNCES such legitimate posts between messages. I'm fairly convinced this parameter should always be 0. If any successful delivery occurs the user should never be removed due to bounces.
What I don't understand is how DEFAULT_MAX_POSTS_BETWEEN_BOUNCES relates to the parameters I see in the admin. None of the parameters in the admin corresponds to this. How is it calculated?
-- greg
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers
-- greg
[Greg Stark]
Before removing a subscriber mailman should send a message with known content testing the address. Only if such a message bounces should a user be dropped.
Uhm... what parts of such a known content message do you think can safely be assumed to still be discernible when Mailman receives the bounce message?
Remember to take the real world into account here; IMO Mailman can't require bounce messages to be formatted in this-or-that particular fashion merely because that format is "common".
Harald
Harald Meland <harald.meland@usit.uio.no> writes:
[Greg Stark]
Before removing a subscriber mailman should send a message with known content testing the address. Only if such a message bounces should a user be dropped.
Uhm... what parts of such a known content message do you think can safely be assumed to still be discernible when Mailman receives the bounce message?
You've misunderstood me. I'm not suggesting it should expect the bounce message to have known content.
What I'm suggesting is that Mailman should *send* a message with known content itself, and only if that message bounces should it decide the address is invalid.
This is what ezmlm does. As much as I dislike ezmlm and qmail for other reasons and like Mailman for other reasons, this is one thing it gets right and Mailman gets wrong.
Deciding an address is invalid on the basis of messages posted to the list is bogus. Mailman can't know whether the message posted to the list bounced because the address was invalid, or merely because the content of that particular message triggered a content-based filter.
-- greg
On Fri, 2003-09-26 at 11:41, Greg Stark wrote:
What I'm suggesting is that Mailman should *send* a message with known content itself, and only if that message bounces should it decide the address is invalid.
It seems difficult to test a negative (what? it doesn't bounce after 10 days? I guess it'll never bounce). I prefer Mailman's positive test approach of sending several notices and requiring an explicit confirmation for reinstatement.
This is what ezmlm does. As much as I dislike ezmlm and qmail for other reasons and like Mailman for other reasons, this is one thing it gets right and Mailman gets wrong.
Deciding an address is invalid on the basis of messages posted to the list is bogus. Mailman can't know whether the message posted to the list bounced because the address was invalid, or merely because the content of that particular message triggered a content-based filter.
Bounce messages triggered by content-based filters are evil and must be eradicated. When SoBig.F came out, we had effective filters in place within a day or so for the specific viruses themselves. What absolutely killed us was all the "helpful" bounces that the zillions of content filters send when they block such a message. And even if you think /that's/ okay, not putting limits on those block messages is still evil. -Barry
Barry Warsaw <barry@python.org> writes:
On Fri, 2003-09-26 at 11:41, Greg Stark wrote:
What I'm suggesting is that Mailman should *send* a message with known content itself, and only if that message bounces should it decide the address is invalid.
It seems difficult to test a negative (what? it doesn't bounce after 10 days? I guess it'll never bounce). I prefer Mailman's positive test approach of sending several notices and requiring an explicit confirmation for reinstatement.
That sounds great, except I'm subscribed to 183 lists, mostly low volume. Periodically I get interested in some project I put aside long ago, check my mail folder for it and discover I've stopped receiving messages months ago. That sucks.
I don't see why using a message with known content is any more of a "negative" test than basing the decision on list messages with unknown content. If you get a bounce from a probe you disable the recipient. That's as much a positive test as looking for bounces.
Deciding an address is invalid on the basis of messages posted to the list is bogus. Mailman can't know whether the message posted to the list bounced because the address was invalid, or merely because the content of that particular message triggered a content-based filter.
Bounce messages triggered by content-based filters are evil and must be eradicated.
As it turns out in this case the mail server isn't directly generating a bounce, it's returning an SMTP 5xx error. So the previous SMTP server in the chain generates a proper bounce to the envelope address.
I've discussed this with the mail admins and their position is that the filters aren't perfect and have caught valid mail in the past. They're worried that simply bitbucketing the mail will cause valid mail to be lost.
Causing valid bounces to be sent to the envelope sender means the innocent victim in the forged From header isn't harassed with bounce messages. Instead the list that resent the message gets the bounce and processes it as a failed message -- which in fact is exactly what happened. It just shouldn't extrapolate from those messages to assuming the mailbox itself is down.
When SoBig.F came out, we had effective filters in place within a day or so for the specific viruses themselves. What absolutely killed us was all the "helpful" bounces that the zillions of content filters send when they block such a message. And even if you think /that's/ okay, not putting limits on those block messages is still evil.
Most of this only applies to the stupid filters that incorrectly send bounce messages to the From header, and often do it with a valid envelope sender themselves so they're just asking for mail loops.
-- greg
On Fri, 2003-09-26 at 12:45, Greg Stark wrote:
That sounds great, except I'm subscribed to 183 lists, mostly low volume. Periodically I get interested in some project I put aside long ago, check my mail folder for it and discover I've stopped receiving messages months ago. That sucks.
Agreed.
I don't see why using a message with known content is any more of a "negative" test than basing the decision on list messages with unknown content. If you get a bounce from a probe you disable the recipient. That's as much a positive test as looking for bounces.
Let me see if I understand your proposal then:
Once your bounce score reaches a certain level, you don't get disabled, but you get on a "hello? are you there" list.
We probe your address for a while, and if we get a bounce, then we disable you and do the normal notifications for reinstatement.
If we /don't/ get a bounce from you for, what? X number of days, what happens? We send another probe and don't disable you? The probe could contain a message similar to the reinstatement message so that you could stop the probes, but otherwise, how long do we keep sending probes? I don't necessarily trust that we can send only one, but OTOH I don't want to fill up your inbox with probe messages.
-Barry
Barry Warsaw <barry@python.org> writes:
- We probe your address for a while, and if we get a bounce, then we disable you and do the normal notifications for reinstatement.
I don't understand this one. Why would you have to poll to check for bounces. You handle the bounce as it comes in.
- If we /don't/ get a bounce from you for, what? X number of days, what
If you don't get a bounce you don't do anything. Do you normally do things when nothing happens?
I don't necessarily trust that we can send only one, but OTOH I don't want to fill up your inbox with probe messages.
Why would you keep sending probes? You only send probes when the conditions are met which are the current conditions for disabling a user.
So what I'm suggesting is:
. When you get a bounce from a list message you record it and see if the limits have been reached. If N consecutive previous list messages to the address have bounced covering the minimum time period etc then you send a probe.
. When you get a bounce from a probe you always disable the address.
You could add a check to see if the probe was within the last few days and check whether any list messages have gotten through since the probe was sent, but all of that is just for paranoia's sake, in case someone flushes an old mail spool after fixing a mailbox.
Actually ezmlm takes that one step further. If a probe bounces then it waits some time period and sends another probe. If the second probe bounces then you're dropped.
-- greg
"Greg" == Greg Stark "Re: [Mailman-Developers] Re: Bounce removal parameters default values" 26 Sep 2003 12:45:46 -0400
Greg> Barry Warsaw <barry@python.org> writes:
Greg> Causing valid bounces to be sent to the envelope senderIs required by the standards.
Greg> Instead the list that resent the message gets the bounce and
Greg> processes it as a failed message -- which in fact is exactly
Greg> what happened.The mailing list is the envelope sender of mail sent to the list. For example the envelope sender of the mail to which this is a reply is
mailman-developers-bounces@python.orgwhich is correct and compliant with the standards. A standards compliant MTA will send any DSN (Delivery Status Notice) to that address.
Greg> It just shouldn't extrapolate from those messages to
Greg> assuming the mailbox itself is down.If any mail is rejected or bounced (ie, initially accepted for delivery but later a DSN is returned indicating a delivery failure) then that is a delivery failure. If you do not like what your receiving mail systems reject or bounce that is not a Mailman problem.
jam"John A. Martin" <jam@jamux.com> writes:
If any mail is rejected or bounced (ie, initially accepted for delivery but later a DSN is returned indicating a delivery failure) then that is a delivery failure. If you do not like what your receiving mail systems reject or bounce that is not a Mailman problem.
I like very much that the mail systems reject virus and worm mails. I don't like that mailman extrapolates from that failure to assuming the mailbox is broken and it should unsubscribe it. That's bogus.
Mailman should not take any such drastic action purely on the basis of a bounce from a message with content it didn't control. It has no idea *why* the message bounced and no idea whether it means future messages will bounce or not.
-- greg
On Thu, 23 Oct 2003, Greg Stark wrote:
I like very much that the mail systems reject virus and worm mails.
That's silly. You should instead like very much that mail clients weren't susceptible to such things and the delivery mechanism didn't have to coddle the mail clients.
Mailman should not take any such drastic action purely on the basis of a bounce
That's the whole point of bounce processing. A bounce signifies an invalid email address. If you don't want bounces to ever cause people to be removed from your mailing lists, turn off bounce processing. If you don't want real messages to get bounced, encourage people to use mail clients that aren't so full of holes that the host mail system needs to cause valid email addresses to bounce.
Dale Newfield <Dale@Newfield.org>
"They that can give up essential liberty to obtain a little safety deserve neither liberty nor safety." - Benjamin Franklin, on the Statue of Liberty
Dale Newfield <Dale@Newfield.org> writes:
On Thu, 23 Oct 2003, Greg Stark wrote:
I like very much that the mail systems reject virus and worm mails.
That's silly. You should instead like very much that mail clients weren't susceptible to such things and the delivery mechanism didn't have to coddle the mail clients.
Mailman should not take any such drastic action purely on the basis of a bounce
That's the whole point of bounce processing. A bounce signifies an invalid email address.
No, bounces can mean various things. Anything from an overfull mailbox, to a message that is too large or otherwise unacceptable. It could be a temporary situation, or it could be because of the particular message.
In any case trusting a message provided from an outside source to serve as a valid test violates security principles. What if i find a message that causes postfix to core dump? I can send it to the mailing list a few times in a row and cause every subscriber of yours using postfix to be unsubscribed from your mailing list.
If you don't want bounces to ever cause people to be removed from your mailing lists, turn off bounce processing.
I'm not the list admin, I'm a poor hapless list subscriber. I get unsubscribed from mailman mailing lists every few months due to this behaviour.
I don't get unsubscribed from ezmlm lists because (as much as I dislike qmail and ezmlm in general) this is one thing it gets right. If ezmlm notices bounces of list messages it doesn't just unsubscribe you summarily, it sends a message of its own with known content and format and only unsubscribes you if that bounces. In fact it does a second iteration of that, which is a good idea but doesn't really seem necessary.
In fact if it weren't for ezmlm's handling of this I would never have figured out why I kept getting dropped from mailman lists. I would have always just assumed it as a bug with mailman.
If you don't want real messages to get bounced
No real messages to me have ever been bounced to my knowledge.
encourage people to use mail clients that aren't so full of holes that the host mail system needs to cause valid email addresses to bounce.
I would love an option to mailman to refuse subscriptions from a list of blacklisted MUAs. I would recommend some lists exclude Outlook on security concerns. It wouldn't reduce the need for proper safe bounce handling. Trusting bounces to messages of unknown content is simply unsafe.
-- greg
On Thu, 2003-10-23 at 10:20, Greg Stark wrote:
"John A. Martin" <jam@jamux.com> writes:
If any mail is rejected or bounced (ie, initially accepted for delivery but later a DSN is returned indicating a delivery failure) then that is a delivery failure. If you do not like what your receiving mail systems reject or bounce that is not a Mailman problem.
I like very much that the mail systems reject virus and worm mails. I don't like that mailman extrapolates from that failure to assuming the mailbox is broken and it should unsubscribe it. That's bogus.
Mailman should not take any such drastic action purely on the basis of a bounce from a message with content it didn't control. It has no idea *why* the message bounced and no idea whether it means future messages will bounce or not.
I've been swamped, but I'll just quickly chime in that we've seen lots of unintentional unsubs since moving python-list over to Mailman 2.1.3. Unintentional means that the person's mailbox is still valid, and they still want to be on the list, but they got disabled without understanding why. I consider it important to fix this for 2.1.4, although I haven't decided how yet. One thing will be to include a bounce example with re-enable notifications. A second thing may be to send probes when the bounce threshold has been reached, but I need to think more about the exact machinery for that and whether that's appropriate for a patch release or not.
-Barry
On Wed, 2003-09-24 at 11:57, Greg Stark wrote:
I should not be removed from a mailing list purely on the basis of bounces of uncontrolled messages. The messages that bounced could have been spam or outlook worms or whatever.
In the default configuration, you won't be. You might get /disabled/ but you won't get removed, unless your MTA also bounces the 3 notices that Mailman sends out at regular intervals to warn disabled users of their subscription status.
Before removing a subscriber mailman should send a message with known content testing the address. Only if such a message bounces should a user be dropped.
That's what it does. The default is to send 3 notices over the course of 3 weeks. Each of those notices contains a url you need to click on to re-instate your subscription. We can't test for a negative (i.e. that the notices don't bounce) so you need to take explicit action to get re-enabled.
As it is I'm being removed from mailing lists whenever a new Outlook worm appears and sends multiple messages in a row. Or a new spammer discovers a list I'm on and sends multiple messages in a row to the list.
It's especially bad on low-volume lists where it's quite possible for spam or Outlook worm messages to be the only messages for days.
Admins of low volume lists might want to change some of the bounce processing defaults. However, by default if a list gets no bounces from you in 7 days, it considers any previous bounce info to be stale and throws it away. So the list would need to get one bounce per day from you for 5 days in a row for you to get disabled.
The other option I would suggest is that list owners start turning on personalization, in order to take advantage of VERP. I think it's harder to get spoof-disabled when VERP is enabled because then Mailman looks for the bouncing address in the encoded recipient header.
But I'm open to specific suggestions for improvement. -Barry
On Friday 26 September 2003 17:47, Barry Warsaw wrote:
Admins of low volume lists might want to change some of the bounce processing defaults. However, by default if a list gets no bounces from you in 7 days, it considers any previous bounce info to be stale and throws it away. So the list would need to get one bounce per day from you for 5 days in a row for you to get disabled.
You write it as it was extremely difficult... but in reality on the server I manage it happened a dozen times in the last month (4 times to me alone). This is getting pretty annoying. I get disabled on 20-30 lists, spread on different virtual domains, so even the "globally" option in the re-enabling form doesn't help much (cause globally = virtual-domain-wide). I don't have any solution but please don't dismiss this problem.
The other option I would suggest is that list owners start turning on personalization, in order to take advantage of VERP. I think it's harder to get spoof-disabled when VERP is enabled because then Mailman looks for the bouncing address in the encoded recipient header.
This looks promising. Is there any howto explaining how to migrate to VERP on a running list and what gotchas to pay attention?
-- Adde parvum parvo magnus acervus erit -- Ovidio
On Fri, 2003-09-26 at 12:24, Simone Piunno wrote:
On Friday 26 September 2003 17:47, Barry Warsaw wrote:
Admins of low volume lists might want to change some of the bounce processing defaults. However, by default if a list gets no bounces from you in 7 days, it considers any previous bounce info to be stale and throws it away. So the list would need to get one bounce per day from you for 5 days in a row for you to get disabled.
You write it as it was extremely difficult... but in reality on the server I manage it happened a dozen times in the last month (4 times to me alone). This is getting pretty annoying. I get disabled on 20-30 lists, spread on different virtual domains, so even the "globally" option in the re-enabling form doesn't help much (cause globally = virtual-domain-wide). I don't have any solution but please don't dismiss this problem.
I won't, and I'm definitely interested in specific suggestions, but of course there are other contributing factors to the problem. Are there better defaults we could be shipping Mailman with that would make the problem less severe without making automatic bounce processing useless?
The other option I would suggest is that list owners start turning on personalization, in order to take advantage of VERP. I think it's harder to get spoof-disabled when VERP is enabled because then Mailman looks for the bouncing address in the encoded recipient header.
This looks promising. Is there any howto explaining how to migrate to VERP on a running list and what gotchas to pay attention?
Part of setting up VERP is MTA specific, and there are instructions in README.POSTFIX and README.EXIM covering those MTAs. Once that's done, its a matter of setting the various VERP_* variables in mm_cfg.py to your tastes, and then turning on personalization (not full) in Non-Digest deliveries.
The bigger question is what the impact of VERPing is on bandwidth and performance. Other than doing the math, I don't have much to add, except that I've started to enable it on most python.org lists and it doesn't seem to be a problem.
-Barry
účastníci (6)
-
Barry Warsaw -
Dale Newfield -
Greg Stark -
Harald Meland -
John A. Martin -
Simone Piunno