Many users unsubscribed at once (not by me)
Today I received 18 unsubscribe notification mesages, each one telling me that "name@addr.ess has been removed from my_list_name". They all appear to have been sent simultaneously, and since I (the list adminisistrator) haven't performed the unsubscribe request myself I suspect either some sort of automated request by the Mailman software or someone hacking into the system. I've had this happen to me once before, several months ago.
The only thing I can think of apart from someone hacking into the list-server is that Mailman automatically removes members who have non-working addresses after a certain amount of list postings. Could this be it, or am I right to be worried and should contact the server owner to ask if an intrusion has shown up on the logs? Logging into the Mailman admin page for my list tells me I'm using Mailman 2.1.12.
Hal
On 12/12/2015 01:46 PM, Hal wrote:
Today I received 18 unsubscribe notification mesages, each one telling me that "name@addr.ess has been removed from my_list_name". They all appear to have been sent simultaneously, and since I (the list adminisistrator) haven't performed the unsubscribe request myself I suspect either some sort of automated request by the Mailman software or someone hacking into the system.
If they were sent at 09:00 server time (that's the default, but it could be different) They were almost certainly unsubscribes done by Mailman's cron/disabled job.
This is a part of Mailman's automated bounce processing.
The only thing I can think of apart from someone hacking into the list-server is that Mailman automatically removes members who have non-working addresses after a certain amount of list postings. Could this be it, or am I right to be worried and should contact the server owner to ask if an intrusion has shown up on the logs?
You should review the settings (and Details links) in your list's web admin Bounce processing section.
Beyond that, if 18 members were unsubscribed at once it is likely that the bounces that caused their delivery to be disabled and them to ultimately be removed were due to something other that the addresses being invalid, e.g., DMARC or possibly some network issue.
See the FAQ article at http://wiki.list.org/x/17891458.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/12/2015 23:04, Mark Sapiro wrote:
On 12/12/2015 01:46 PM, Hal wrote:
Today I received 18 unsubscribe notification mesages, each one telling me that "name@addr.ess has been removed from my_list_name".
If they were sent at 09:00 server time (that's the default, but it could be different) They were almost certainly unsubscribes done by Mailman's cron/disabled job.
Thanks -Yes, that's right! 09.00 at the server's time.
This is a part of Mailman's automated bounce processing.
Glad to have this sorted out!
You should review the settings (and Details links) in your list's web admin Bounce processing section.
Sure. Here are my bounce settings (I believe they're the default ones though as I try not to touch stuff I don't fully understand):
bounce processing: YES bounce score threshold: 5.0 bounce info state after: 7 bounce you are disabled warnings: 3 bounce you are disabled interval: 7
bounce unrecognized goes to list owner: YES bounce notify owner on disable: YES bounce notify owner on removale: YES
Beyond that, if 18 members were unsubscribed at once it is likely that the bounces that caused their delivery to be disabled and them to ultimately be removed were due to something other that the addresses being invalid, e.g., DMARC or possibly some network issue.
See the FAQ article at http://wiki.list.org/x/17891458.
Thanks. So I understand that a certain procedure for fighting spam has been set up by many email providers but this is incompatible with how Mailman works, causing problems such as the ones I'm experiencing.
Seeing that 17 out of those 18 unsubscribed messages were sent from Hotmail I can only assume the above was what happened. Next I will locate the other instance I had of automated removals to see if there's a similar pattern there.
Most of the discussions in the FAQ article were over my head, so I suppose to make it simple I should just change the "bounce processing" option to OFF, then resubscribe all the members which were unsubscribed and finally emailing each of them with an apology and explanation.
So, by turning this option off I will lose Mailman's ability to automatically remove addresses which are no longer reachable? What else will I be missing out on?
Hal
of yesterday I found that 17 of them were from hotmail. The remaining message was from live.com.au. I will try and find the other instance (several months ago) of Mailmain As there seems
Most of that stuff was way over my head but I did understand that
On 13/12/2015 11:14, Hal wrote:
On 12/12/2015 23:04, Mark Sapiro wrote:
On 12/12/2015 01:46 PM, Hal wrote:
Today I received 18 unsubscribe notification mesages, each one telling me that "name@addr.ess has been removed from my_list_name".
If they were sent at 09:00 server time (that's the default, but it could be different) They were almost certainly unsubscribes done by Mailman's cron/disabled job.
Thanks -Yes, that's right! 09.00 at the server's time.
Seeing that 17 out of those 18 unsubscribed messages were sent from Hotmail I can only assume the above was what happened. Next I will locate the other instance I had of automated removals to see if there's a similar pattern there.
I found the previous instance (actually a couple of other instances) when list members had automatically been removed at 09:00, but I didn't see any pattern in the addresses as they were all different. Still, I suppose I should just turn the automatic removal option off so this won't ever happen again.
Hal
Hi,
Unfortunately, disabling automatic removal from the list will only fix one effect of the problem, not the problem itself. There's something going on which is causing mail to those addresses to bounce. If you are receiving messages with the subject of "Bounce action notification" these will tell you who is bouncing mail, and the attached mail delivery failure reports will tell you why they are bouncing mail, which is important. I suspect you have a DMARC or IP reputation issue.
Hope this helps.
Jayson
On 12/13/2015 5:55 AM, Hal wrote:
On 13/12/2015 11:14, Hal wrote:
On 12/12/2015 23:04, Mark Sapiro wrote:
On 12/12/2015 01:46 PM, Hal wrote:
Today I received 18 unsubscribe notification mesages, each one telling me that "name@addr.ess has been removed from my_list_name".
If they were sent at 09:00 server time (that's the default, but it could be different) They were almost certainly unsubscribes done by Mailman's cron/disabled job.
Thanks -Yes, that's right! 09.00 at the server's time.
Seeing that 17 out of those 18 unsubscribed messages were sent from Hotmail I can only assume the above was what happened. Next I will locate the other instance I had of automated removals to see if there's a similar pattern there.
I found the previous instance (actually a couple of other instances) when list members had automatically been removed at 09:00, but I didn't see any pattern in the addresses as they were all different. Still, I suppose I should just turn the automatic removal option off so this won't ever happen again.
Hal
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/jaybird%40bluegrasspal...
On 13/12/2015 13:05, Jayson Smith wrote:
Hi,
Unfortunately, disabling automatic removal from the list will only fix one effect of the problem, not the problem itself. There's something going on which is causing mail to those addresses to bounce. If you are receiving messages with the subject of "Bounce action notification" these will tell you who is bouncing mail, and the attached mail delivery failure reports will tell you why they are bouncing mail, which is important. I suspect you have a DMARC or IP reputation issue.
I'm not sure I fully understand this and what to do about it.
Are you saying that even though I turn "automatic removal" off I will still have postings from certain list members NOT reaching the list (but bouncing with an error message back to me, the list owner -possibly also with an error alert to the poster as well)?
Are there other restrictions in the Mailman software which stops postings based on other criteria? If yes, can this/these features be turned off so in the end I only rely on "only allow posting from existing members"? I assume having the latter enabled will at least prevent automated spam messages from reaching the list.
Alternatively I'm guessing the solution is to set the list to moderation and approve every single message manually. I do have a relatively low-volume list, so this is doable (but not something I would enjoy enabling) -how is this done actually? By logging into the administration interface and clicking on something for every message, or can it be done simpler/quicker by simply pressing reply to a certain admin message or something?
Hal
Hi,
The problem isn't with Mailman receiving unwanted messages. The problem is that Mailman is trying to deliver outbound messages, and is having problems, messages are being returned as undeliverable. Everything in Mailman is working properly up to the point where messages are being bounced. This could be for a number of reasons. Some users might have outdated addresses, in which case messages would be bounced back with errors similar to "No such user" or "User unknown." Some people may have full mailboxes. But the more likely problem, since you're having many issues come up all at once is that there's a problem between your server and the destination mail server for those messages. It could be that the DMARC settings in Mailman aren't set correctly and those servers are rejecting messages they don't like, or it could be that your server's IP address has somehow received a bad reputation, and certain mail servers will reject all mail from your server, regardless of sender, recipient, or content.
Jayson
On 12/13/2015 7:42 AM, Hal wrote:
On 13/12/2015 13:05, Jayson Smith wrote:
Hi,
Unfortunately, disabling automatic removal from the list will only fix one effect of the problem, not the problem itself. There's something going on which is causing mail to those addresses to bounce. If you are receiving messages with the subject of "Bounce action notification" these will tell you who is bouncing mail, and the attached mail delivery failure reports will tell you why they are bouncing mail, which is important. I suspect you have a DMARC or IP reputation issue.
I'm not sure I fully understand this and what to do about it.
Are you saying that even though I turn "automatic removal" off I will still have postings from certain list members NOT reaching the list (but bouncing with an error message back to me, the list owner -possibly also with an error alert to the poster as well)?
Are there other restrictions in the Mailman software which stops postings based on other criteria? If yes, can this/these features be turned off so in the end I only rely on "only allow posting from existing members"? I assume having the latter enabled will at least prevent automated spam messages from reaching the list.
Alternatively I'm guessing the solution is to set the list to moderation and approve every single message manually. I do have a relatively low-volume list, so this is doable (but not something I would enjoy enabling) -how is this done actually? By logging into the administration interface and clicking on something for every message, or can it be done simpler/quicker by simply pressing reply to a certain admin message or something?
Hal
In a message of Sun, 13 Dec 2015 13:42:47 +0100, Hal writes:
On 13/12/2015 13:05, Jayson Smith wrote:
Hi,
Unfortunately, disabling automatic removal from the list will only fix one effect of the problem, not the problem itself. There's something going on which is causing mail to those addresses to bounce. If you are receiving messages with the subject of "Bounce action notification" these will tell you who is bouncing mail, and the attached mail delivery failure reports will tell you why they are bouncing mail, which is important. I suspect you have a DMARC or IP reputation issue.
I'm not sure I fully understand this and what to do about it.
Are you saying that even though I turn "automatic removal" off I will still have postings from certain list members NOT reaching the list (but bouncing with an error message back to me, the list owner -possibly also with an error alert to the poster as well)?
This is possible. What is more likely is that a member of your list, posts something to the list. Then mailman tries to deliver the mail to all the subscribers. When it calls up some_site asking to deliver mail to subscribers who have subscribed from accounts on some_site, some_site says something to the effect of:
Drop dead. We are protecting our users from receiving mail from you! No matter how many times you ask us to send mail to them, we _won't_ because _we don't like you and your crappy mailing list_.
So you need to check your logs and find out if there is a some_site, or more than one some_site which really doesn't like you.
The usual reasons for disliking you are:
- We hate your IP, you have bad reputation with us, maybe you have been reported as a spammer someplace, or maybe you just send us a lot of mail and we don't like that.
Mailman cannot do anything about this problem. Talking to the site that hates you can, if the site that hates you is willing to talk to you at all about the problem. Large sites, like aol typically do not. "We hate you because you are a spammer, and the last thing we want to do is to tell you how to get around our spam protection. We don't care how many innocent non-spammers are hurt by our policies which has falsely caused them to be considered spammers. So if you aren't a spammer, too bad, we won't lose any sleep over the fact that your mail cannot arrive here. And we won't talk to you about it, either."
We have a DMARC policy which is designed as follows:
If mail comes in that originates from a user on one of our sites, and it doesn't come in on one of our servers, we will call this spam and refuse to deliver it.
The big offenders here are aol and yahoo. This policy breaks every mailing list on the planet.
A user on yahoo.com sends mail to a mailing list, and the list tries to send it to all the subscribers, and when it tries for all the other subscribers at yahoo.com, yahoo says, 'This mail that supposedly came from user@yahoo.com, didn't come from one of our servers. Do not deliver.'
If this is your problem, and we think it is, you need a mailman at least 2.16 and 2.18 is better. Then we can tell you how to configure it to rewrite things so that your mailing list mesages no longer appear to come from their senders, so yahoo doesn't reject them, but does let people reply to such messages and have the reply go to the real sender, at yahoo or wherever the way that you expect.
But first you have to figure out why you are getting the bounces in the first place. Check your mailman log files for that. They are in different places depending on whether you have a CPanel mailman, or a regular one, or one of countless other hosting solutions that I am unfamiliar with.
Does this make sense?
Laura Creighton
On 13/12/2015 15:53, Laura Creighton wrote:
What is more likely is that a member of your list, posts something to the list. Then mailman tries to deliver the mail to all the subscribers. When it calls up some_site asking to deliver mail to subscribers who have subscribed from accounts on some_site, some_site says something to the effect of:
Drop dead. We are protecting our users from receiving mail from you! No matter how many times you ask us to send mail to them, we _won't_ because _we don't like you and your crappy mailing list_.
So "some_site" in your example are all the mail servers around the world which REFUSE to receive the posting(s) from my mailing list, and thus sends an error message back to my list server's Mailman software.
And as these error messages on my list-server start piling up from various mail servers around the world (i.e. "some_site") Mailman says "OK, enough! The LIST-MEMBER sending those messages has to be banned as messages from him won't receive a vast majority of the other list members anyway, so I'll remove him so it won't happen again".
Have I more or less understood it correctly?
So you need to check your logs and find out if there is a some_site, or more than one some_site which really doesn't like you.
OK, I'll ask the server-admin/owner about the logs as I'm not sure where they are or how to access them. Which logs specifically are we looking for as I'm sure there are many?
I assumed I would get an error message (email) for every time a posting didn't reach the list but the latest "bounce action notification" I have is from 5 months back, so apparently not. Maybe I've messed with some settings in the Mailman web control panel as I kept getting so many of them and didn't know what to do about it all.
The usual reasons for disliking you are:
- We hate your IP, you have bad reputation with us, maybe you have been reported as a spammer someplace, or maybe you just send us a lot of mail and we don't like that.
Again, this is the IP address of certain LIST-MEMBERS, right, and not my list-server?
Does this also mean that the same receiving mail-servers will refuse email sent from the same LIST-MEMBER, but sent directly (i.e. outside of my mailing list)?
Mailman cannot do anything about this problem. Talking to the site that hates you can, if the site that hates you is willing to talk to you at all about the problem. Large sites, like aol typically do not.
I don't have the capacity nor time to contact every single email provider refusing to receive messages from certain list members.
We have a DMARC policy which is designed as follows:
If mail comes in that originates from a user on one of our sites, and it doesn't come in on one of our servers, we will call this spam and refuse to deliver it.
The big offenders here are aol and yahoo. This policy breaks every mailing list on the planet.
A user on yahoo.com sends mail to a mailing list, and the list tries to send it to all the subscribers, and when it tries for all the other subscribers at yahoo.com, yahoo says, 'This mail that supposedly came from user@yahoo.com, didn't come from one of our servers. Do not deliver.'
Ouch! So it won't accept the email because my list-server PASSES ON a message, and that IP address is obviously not the same as that of the LIST MEMBER posting that message in the first place?
If this is your problem, and we think it is, you need a mailman at least 2.16 and 2.18 is better.
I'll contact my server-owner/admin to upgrade the Mailman software. I see from the Mailman site (http://www.list.org/) that version 2.1.20 is the latest version, and there's version 3.01 as well ("Show don't tell", whatever that means). Is 2.18 the latest stable version?
Then we can tell you how to
configure it to rewrite things so that your mailing list mesages no longer appear to come from their senders, so yahoo doesn't reject them, but does let people reply to such messages and have the reply go to the real sender, at yahoo or wherever the way that you expect.
I know many list-owners prefer to have messages reply back to the poster of the message instead of the list itself (I've read those discussions many times though I forget why), but for my list I'm sure this would mean A LOT of list discussions done outside of the list, which in turn defeats the purpose of my list (sharing information).
Does this make sense?
Certainly more than a couple of days ago! Thanks for explaining :-)
Hal
What is more likely is that a member of your list, posts something to the list. Then mailman tries to deliver the mail to all the subscribers. When it calls up some_site asking to deliver mail to subscribers who have subscribed from accounts on some_site, some_site says something to the effect of:
Drop dead. We are protecting our users from receiving mail from you! No matter how many times you ask us to send mail to them, we _won't_ because _we don't like you and your crappy mailing list_.
So "some_site" in your example are all the mail servers around the world which REFUSE to receive the posting(s) from my mailing list, and thus sends an error message back to my list server's Mailman software.
In this particular example, some_site refers to the specific host the receiving email addresses that were just automatically unsubscribed were on, but it could be extrapolated to any site that is blocking your IP.
And as these error messages on my list-server start piling up from various mail servers around the world (i.e. "some_site") Mailman says "OK, enough! The LIST-MEMBER sending those messages has to be banned as messages from him won't receive a vast majority of the other list members anyway, so I'll remove him so it won't happen again".
No, that isn't quite correct. Mailman sees that messages to subscriber1@some_site are bouncing and unsubscribed subscriber1@some_site, then sees that messages to subscriber2@some_site are bouncing and unsubscribes subscriber2@some_site.
Have I more or less understood it correctly?
The recipient addresses are the ones bouncing, and are therefore the ones that are unsubscribed to stop the bouncing - and to protect your IP addresses reputation by making it look less like you are a spammer.
<snip log location question and bounce notification question>
The usual reasons for disliking you are:
- We hate your IP, you have bad reputation with us, maybe you have been reported as a spammer someplace, or maybe you just send us a lot of mail and we don't like that.
Again, this is the IP address of certain LIST-MEMBERS, right, and not my list-server?
In this case, the IP of the list server.
Does this also mean that the same receiving mail-servers will refuse email sent from the same LIST-MEMBER, but sent directly (i.e. outside of my mailing list)?
If this is the case, the receiving server (some_site) would likely refuse any message sent from your list server, whether it was sent through the mailing list or sent directly.
Mailman cannot do anything about this problem. Talking to the site that hates you can, if the site that hates you is willing to talk to you at all about the problem. Large sites, like aol typically do not.
I don't have the capacity nor time to contact every single email provider refusing to receive messages from certain list members.
They probably aren't refusing messages from certain list members, but every message on your mailing list and possibly every message from your mail server. Of course, this relies on this actually being the issue.
Another possibility is that a school or corporate mail server administrator made a policy change and pushed new email addresses to everyone, then after 4 or 6 or 8 months, or maybe a year of forwarding old addresses to new addresses, deleted the old addresses so they are no longer valid and these bounced generating the rash of unsubscribes. In part, this is done by mailman so you do not look like a spammer who will likely not bother removing invalid addresses from their address lists. This is very similar to what has been going on with the cable-provider-I-can't-remember-the-name-of to Bresnan to ComCast to Charter changes in this part of the US over the last few years. All the email addresses of the precursor to Bresnan have been deleted, and in most areas all personal and many business Bresnan email addresses have been deleted. Some users have had their email address forcibly changed four times in four years...
We have a DMARC policy which is designed as follows:
If mail comes in that originates from a user on one of our sites, and it doesn't come in on one of our servers, we will call this spam and refuse to deliver it.
The big offenders here are aol and yahoo. This policy breaks every mailing list on the planet.
A user on yahoo.com sends mail to a mailing list, and the list tries to send it to all the subscribers, and when it tries for all the other subscribers at yahoo.com, yahoo says, 'This mail that supposedly came from user@yahoo.com, didn't come from one of our servers. Do not deliver.'
Ouch! So it won't accept the email because my list-server PASSES ON a message, and that IP address is obviously not the same as that of the LIST MEMBER posting that message in the first place?
Correct, if this is the issue causing your bounces.
Keith
--
from my mac to yours...
Keith Seyffarth mailto:weif@weif.net http://www.weif.net/ - Home of the First Tank Guide! http://www.rpgcalendar.net/ - the Montana Role-Playing Calendar
http://www.miscon.org/ - Montana's Longest Running Science Fiction Convention
In a message of Sun, 13 Dec 2015 17:11:03 +0100, Hal writes:
On 13/12/2015 15:53, Laura Creighton wrote:
What is more likely is that a member of your list, posts something to the list. Then mailman tries to deliver the mail to all the subscribers. When it calls up some_site asking to deliver mail to subscribers who have subscribed from accounts on some_site, some_site says something to the effect of:
Drop dead. We are protecting our users from receiving mail from you! No matter how many times you ask us to send mail to them, we _won't_ because _we don't like you and your crappy mailing list_.
So "some_site" in your example are all the mail servers around the world which REFUSE to receive the posting(s) from my mailing list, and thus sends an error message back to my list server's Mailman software.
It could be that you are in such a sad state that many places won't deal with you, but usually the problem is one site, where you have many users, and that site won't deal with you. If your 18 bounces are all from different places, well, maybe you do have 18 users who have all managed to use up their disk quota at the same time. Especially if people on your list have started sending really, really large files for some reason.
But if the 18 users are all from the same domain, that is reason to strongly suspect that the reason mail to these users is bouncing is that their site isn't talking to you, rather than their disk quota is full, they all cancelled their accounts and don't exit, they died, they got fired and their accounts were removed or any of these other, legitimate reasons for mail to bounce, which has to do with a problem with their account. Their accounts are fine. You just cannot send mail to them.
And as these error messages on my list-server start piling up from various mail servers around the world (i.e. "some_site") Mailman says "OK, enough! The LIST-MEMBER sending those messages has to be banned as messages from him won't receive a vast majority of the other list members anyway, so I'll remove him so it won't happen again".
Have I more or less understood it correctly?
No. Let us say you have 5 users on your list, user1@aol.com, user2@aol.com, user3@aol.com user4@aol.com and user5@aol.com.
user1@aol.com posts a piece of mail to your list. mailman tries to deliver to user2, user3, user4 and user5 @ aol.com aol says "drop dead, we don't talk to you because of our DMARC policy" mail to user2 user3 user4 and user5 bounces. Their bounce count is incremented.
user2@aol.com posts a piece of mail to your list. mailman tries to deliver to user1, user3, user4 and user5 @ aol.com aol says "drop dead, we don't talk to you because of our DMARC policy" mail to user1 user3 user4 and user5 bounces. Their bounce count is incremented.
and so on. Every time somebody from aol sends mail to the list, it bounces for every other aol member on your list. Their bounce counts increase.
One day, some message sends some of the bounce counts over the limit mailman has, after which it says -- Too many bounces! I cannot deliver mail to this account! Unsubscribe this person! And, because of the way things have happened you get a triggering message which causes a lot of unsubscribes _from the same site_.
This is what we think has happened to you. We would be very pleased to be mistaken about this, but you have to check the logs to find out why the mail is bouncing.
Whether 18 bounce unsubscribes is a lot depends on the size of your mailing list. If your list has 200 members, its an indication something is seriously wrong. If it has 20,000 members then everything may be just fine.
So you need to check your logs and find out if there is a some_site, or more than one some_site which really doesn't like you.
OK, I'll ask the server-admin/owner about the logs as I'm not sure where they are or how to access them. Which logs specifically are we looking for as I'm sure there are many?
I assumed I would get an error message (email) for every time a posting didn't reach the list but the latest "bounce action notification" I have is from 5 months back, so apparently not. Maybe I've messed with some settings in the Mailman web control panel as I kept getting so many of them and didn't know what to do about it all.
What sort of mailman do you have? What is the url for the administrative interface? What OS is it running under? I only know of some of the places you can find mailman logs, but if we can find out what system you have and how it is set up and administered, then somebody here will know where to look.
The usual reasons for disliking you are:
- We hate your IP, you have bad reputation with us, maybe you have been reported as a spammer someplace, or maybe you just send us a lot of mail and we don't like that.
Again, this is the IP address of certain LIST-MEMBERS, right, and not my list-server?
No, it is your server.
Ouch! So it won't accept the email because my list-server PASSES ON a message, and that IP address is obviously not the same as that of the LIST MEMBER posting that message in the first place?
Yes.
If this is your problem, and we think it is, you need a mailman at least 2.16 and 2.18 is better.
I'll contact my server-owner/admin to upgrade the Mailman software. I see from the Mailman site (http://www.list.org/) that version 2.1.20 is the latest version, and there's version 3.01 as well ("Show don't tell", whatever that means). Is 2.18 the latest stable version?
No. The current stable GNU Mailman version are 2.1.20 released on 31-Mar-2015.
I know many list-owners prefer to have messages reply back to the poster of the message instead of the list itself (I've read those discussions many times though I forget why), but for my list I'm sure this would mean A LOT of list discussions done outside of the list, which in turn defeats the purpose of my list (sharing information).
If you have your list configured so that reply-to goes to the list rather than to the sender, you will have a more difficult time of it, but _things can be done_.
Does this make sense?
Certainly more than a couple of days ago! Thanks for explaining :-)
Hal
Good luck.
Laura
On 12/13/2015 5:14 AM, Hal wrote:
Seeing that 17 out of those 18 unsubscribed messages were sent from Hotmail I can only assume the above was what happened. [etc]
This sounds as if Hotmail has joined the villainous club formerly composed only of Yahoo and AOL, publishing a DMARC policy of "p=reject". Can anyone else confirm that this has happened?
(Despite all the work Mark has put into updating Mailman 2.1.x to deal automatically with the DMARC problem, I'm still handling it by putting Yahoo and AOL users on moderation and manually wrapping their posts. I do this because I'm not sure I can successfully upgrade Mailman on a Plesk system that I don't really understand; and it's feasible because I have only two very low-volume lists. I guess I now have to do the same with Hotmail users?)
-- Larry Kuenning larry@qhpress.org
On 12/13/2015 12:01 PM, Laura Creighton wrote:
Have I more or less understood it correctly?
No. Let us say you have 5 users on your list, user1@aol.com, user2@aol.com, user3@aol.com user4@aol.com and user5@aol.com.
user1@aol.com posts a piece of mail to your list. mailman tries to deliver to user2, user3, user4 and user5 @ aol.com aol says "drop dead, we don't talk to you because of our DMARC policy" mail to user2 user3 user4 and user5 bounces. Their bounce count is incremented.
But this account is still incomplete.
Besides what AOL does with the attempted deliveries to user2, user3, user4, and user5, you also get problems with delivery to users on a lot of other domains. The reason is that some other domains automatically consult AOL to see what to do with mail that claims to come from user1@aol.com but didn't reach them directly from an AOL server. They find that AOL has published a DMARC policy of "p=reject", and they obediently follow AOL's instruction to reject the post. So it bounces not only for the other AOL users (2 through 5) but for a lot of users on miscellaneous other domains.
The problem originates with AOL's DMARC policy but creates bounces in delivery attempts to users on many other domains because those domains have decided to respect AOL's published "p=reject" policy.
And Hal's report suggests that this is now happening not only with AOL and Yahoo (which started this practice in April 2014) but with messages originating on Hotmail as well. Can anyone check that Hotmail has published a "p=reject" DMARC policy?
-- Larry Kuenning larry@qhpress.org
On 12/13/2015 02:14 AM, Hal wrote:
Sure. Here are my bounce settings (I believe they're the default ones though as I try not to touch stuff I don't fully understand):
bounce processing: YES bounce score threshold: 5.0 bounce info state after: 7 bounce you are disabled warnings: 3 bounce you are disabled interval: 7
bounce unrecognized goes to list owner: YES bounce notify owner on disable: YES bounce notify owner on removale: YES
Yes, these are the defaults. There are a number of other replies in this thread and I didn't thoroughly read them all, so some of this may be redundant, but with the above settings (ignoring typos) if a list member's ISP bounces posts on 5 different days with no more than 7 days between bounces, that member's delivery will be disabled and a notice sent to the list owner (bounce notify owner on disable: = YES) containing a copy of the bounce message from the member's ISP. This is where you find out why the ISP is rejecting the message.
It may be because of things like "no such user here" in which case, you need do nothing more because bounce processing is doing what it's intended to do. The list member will be sent a notice as well which will also bounce for the same reason, but that's OK. After 3 7-day intervals, the member will be unsubscribed and the owner notified.
If the reason in the disabling notice is something else like "message rejected for DMARC policy reasons" or "we don't like the sending server", these are things where you may take additional actions. There are two basic classes here. I'll take them separately.
Rejects because the recipient ISP doesn't like the sending server can be addressed, but only by the people who control the server. Things like ensuring proper full-circle DNS (See https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS), DKIM signing outgoing mail, publishing SPF and signing up for various ISP monitoring services may help. See http://wiki.list.org/x/4030690.
Rejects because of DMARC policy occur because the domain of the author of the post publishes a DMARC p=reject policy, E.g., the post that is bounced is From: some_user@yahoo.com. Without the list applying some DMARC mitigation, this post will be bounced by every ISP (not just Yahoo and AOL) that honors the published DMARC policy of the From: domain. Note that in this case, the list member should receive the warning notices sent to the member, and should be able to re-enable delivery and avoid being unsubscribed.
Beginning with Mailman 2.1.16, and much improved in Mailman 2.1.18, there are list settings that help with DMARC mitigation by altering the From: header of the post or wrapping the post in an outer message From: the list. Prior to Mailman 2.1.16 the only thing a list owner can do are items 2), 3), 6) or 7) in the FAQ at http://wiki.list.org/x/17891458.
The bottom line is look at the notices you receive when a member's delivery is first disabled by bounce and figure out why.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/13/2015 09:49 AM, Larry Kuenning wrote:
On 12/13/2015 5:14 AM, Hal wrote:
Seeing that 17 out of those 18 unsubscribed messages were sent from Hotmail I can only assume the above was what happened. [etc]
This sounds as if Hotmail has joined the villainous club formerly composed only of Yahoo and AOL, publishing a DMARC policy of "p=reject". Can anyone else confirm that this has happened?
This is not the case. First of all, it doesn't say anything about hotmail's DMARC policy. It only says that if these are DMARC bounces, hotmail is honoring published DMARC policies.
In any case, here's hotmail.com's DMARC record.
_dmarc.hotmail.com. 3600 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\; fo=1"
p=none means no DMARC policy. This only means that ISP's should not check DMARC for mail From: the hotmail.com domain. It does not mean that hotmail won't check DMARC on incoming mail From: domains that do publish a DMARC policy other than none, and in fact hotmail and other Microsoft server do check DMARC and have from the beginning.
You can find the DMARC policy for any domain, e.g. example.com, by querying DNS for a TXT record for the domain with _dmarc. prepended, e.g. _dmarc.example.com.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/13/2015 10:35 AM, Larry Kuenning wrote:
And Hal's report suggests that this is now happening not only with AOL and Yahoo (which started this practice in April 2014) but with messages originating on Hotmail as well.
No. Hal's report only said that most of the unsubscribed user's were hotmail. It said nothing about the From: domain of the posts that were bounced.
In fact, the fact that almost all the unsubscribed users were hotmail makes it seem that this is not DMARC, but more likely hotmail (Microsoft) blocking the sending IP.
Can anyone check that Hotmail has published a "p=reject" DMARC policy?
See my reply to your other post https://mail.python.org/pipermail/mailman-users/2015-December/080216.html.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/13/2015 08:11 AM, Hal wrote:
I assumed I would get an error message (email) for every time a posting didn't reach the list but the latest "bounce action notification" I have is from 5 months back, so apparently not. Maybe I've messed with some settings in the Mailman web control panel as I kept getting so many of them and didn't know what to do about it all.
You don't get a notice for every bounce. With all the notices enabled as you posted earlier, you get every message sent to the list-bounces address which Mailman can't parse as a bounce. Normally, this is only spam. You also get only one "bounce action notification" containing a copy of the disabling bounce when Mailman disables delivery to a member for bouncing, and you get the "unsubscribed" notice when the member is removed.
In this case, you should have received the "bounce action notification" messages for each of the unsubscribed members three weeks prior to their being unsubscribed. If you didn't receive these, there is some problem with their delivery. Maybe they went to a spam folder, maybe your ISP rejected or discarded them. The outgoing mail logs on the Mailman server will have more info if you can convince someone to look at them and they know what to look for and they haven't been rotated out of existence by now.
The usual reasons for disliking you are:
- We hate your IP, you have bad reputation with us, maybe you have been reported as a spammer someplace, or maybe you just send us a lot of mail and we don't like that.
Again, this is the IP address of certain LIST-MEMBERS, right, and not my list-server?
No. Laura is talking about the IP address of the Mailman server.
Does this also mean that the same receiving mail-servers will refuse email sent from the same LIST-MEMBER, but sent directly (i.e. outside of my mailing list)?
No.
I'll contact my server-owner/admin to upgrade the Mailman software. I see from the Mailman site (http://www.list.org/) that version 2.1.20 is the latest version, and there's version 3.01 as well ("Show don't tell", whatever that means). Is 2.18 the latest stable version?
No, 2.1.20 is the latest 'stable' version of the 2.1 series. Mailman 3 is a major change and is currently recommended only for new lists as there is as yet no defined migration path for MM 2.1 lists.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/13/2015 04:42 AM, Hal wrote:
Are you saying that even though I turn "automatic removal" off I will still have postings from certain list members NOT reaching the list (but bouncing with an error message back to me, the list owner -possibly also with an error alert to the poster as well)?
If you turn of bounce_processing (the only way to turn off automatic removal other than perhaps setting bounce_you_are_disabled_warnings to a very large number), Mailman will ignore all bounces of posts. It won't notify the admin or the user, and if the bouncing user's ISP is compliant, it won't notify the poster either as all bounce notifications should be returned to the envelope sender of the delivered post which is Mailman and not the poster.
Are there other restrictions in the Mailman software which stops postings based on other criteria? If yes, can this/these features be turned off so in the end I only rely on "only allow posting from existing members"? I assume having the latter enabled will at least prevent automated spam messages from reaching the list.
As long as the spam doesn't spoof a list member address.
Alternatively I'm guessing the solution is to set the list to moderation and approve every single message manually. I do have a relatively low-volume list, so this is doable (but not something I would enjoy enabling) -how is this done actually? By logging into the administration interface and clicking on something for every message, or can it be done simpler/quicker by simply pressing reply to a certain admin message or something?
Controlling what posts reach the list will not have any effect on posts which are blocked by recipient ISPs for DMARC or other policy reasons except perhaps indirectly by affecting the reputation of the Mailman server, but if I understand correctly, your list is one of many on a shared host, so you are in some sense at the mercy of others.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 13 Dec 2015, at 13:56, Mark Sapiro wrote:
In fact, the fact that almost all the unsubscribed users were hotmail makes it seem that this is not DMARC, but more likely hotmail (Microsoft) blocking the sending IP.
And with the one other being live.com.au (roughly: Australian Hotmail) it becomes a distinctively Microsoft problem.
For what it is worth, recently the policy staff and policies of Microsoft's former-Hotmail and former-Frontbridge have seemed to be converging/merging. One win for the world of this is that the chronic problem of Hotmail/Live.com mail being accepted but silently discarded has been replaced by overt rejection at the MX border (no, not THAT MX border) in many cases. For Mailman lists, one result is that long-dead addresses are getting cleaned out because they are now bouncing. Another result is that some silent discards have been replaced by delivery, so in some cases people who haven't seen mail from a list in years and now are seeing it and telling MS that it's spam. The second process can lead to broad blocking across MS-hosted domains, but at least now it is visible.
participants (7)
-
Bill Cole -
Hal -
Jayson Smith -
Keith Seyffarth -
Larry Kuenning -
Laura Creighton -
Mark Sapiro