Mailman message IDs? Disappearing Posts
What does mailman do to message IDs? When a message is posted to list subscribers, does it have the same message ID, or a predictably modified message ID that I can grep for?
I'm trying to trace disappearing posts from a couple of list members. Their posts go through postfix, mailman reports the posts as succesfully posted, and the posts show up in the list archives, but not in the subscribers mailboxes.
I have a small list with about 20 members. Most of the members, the list works fine; mail is showing up, being sent out, being received and replied to, etc.
A handful of members lots of bounces, probably due to their provider's policies (yahoo and hotmail, although *sometimes* the messages get through and one yahoo user has posted successfully).
However, for these two yahoo subscribers, their posts get reported as posted successfully, but are never received by the list subscribers.
I grep postfix logs (/var/log/mail.log) for the sender's email address, I see a message coming in, and postfix handing it off to mailman post (full log excerpt at end of message):
Jul 9 07:52:53 ip-172-31-8-246 postfix/local[5175]: 4A11D26A33: to=<Examplelist@example.com>, relay=local, delay=0.46, delays=0.36/0.01/0/0.09, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post examplelist)
I grep that message ID in /var/log/mailman/* and I see one line, telling me it was posted successfully:
post:Jul 09 07:52:54 2014 (4216) post to listname from example@yahoo.com, size=3411, message-id=<1404892372.90900.YahooMailBasic@web163105.mail.bf1.yahoo.com>, success
No other occurrences of that message ID in /var/log/mailman/*. Or anywhere in /var/log other than the lines of postfix receiving it, forwarding it and cleaning it up (see log excerpt at end).
The mailman post log just says "posted successfully", nothing else in there to indicate where to look next, or what the new message ID (if any) might be. Any hints?
Grepping for lines with a similar timestamp turns up half a dozen bounce messages two seconds later, which may explains why those 6 subscribers don't get the messages, but not the rest.
/var/log/mailman/error is empty.
/var/lib/mailman/qfiles subdirectories are empty.
Content filtering is NOT on. Though I just went in and changed the action from "discard" to "forward to list owner" (me), just in case.
--/var/log/mail.log--------------------------------------------------- Jul 9 07:52:52 ip-172-31-8-246 postfix/smtpd[5171]: connect from nm12-vm0.bullet.mail.bf1.yahoo.com[98.139.213.140] Jul 9 07:52:53 ip-172-31-8-246 postfix/smtpd[5171]: 4A11D26A33: client=nm12-vm0.bullet.mail.bf1.yahoo.com[98.139.213.140] Jul 9 07:52:53 ip-172-31-8-246 postfix/cleanup[5174]: 4A11D26A33: message-id=<1404892372.90900.YahooMailBasic@web163105.mail.bf1.yahoo.com> Jul 9 07:52:53 ip-172-31-8-246 postfix/qmgr[1119]: 4A11D26A33: from=<exampleuser@yahoo.com>, size=2415, nrcpt=1 (queue active) Jul 9 07:52:53 ip-172-31-8-246 postfix/local[5175]: 4A11D26A33: to=<Examplelist@example.com>, relay=local, delay=0.46, delays=0.36/0.01/0/0.09, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post examplelist) Jul 9 07:52:53 ip-172-31-8-246 postfix/qmgr[1119]: 4A11D26A33: removed Jul 9 07:52:53 ip-172-31-8-246 postfix/smtpd[5171]: disconnect from nm12-vm0.bullet.mail.bf1.yahoo.com[98.139.213.140] Jul 9 07:52:54 ip-172-31-8-246 postfix/smtpd[5171]: connect from localhost[127.0.0.1] Jul 9 07:52:54 ip-172-31-8-246 postfix/smtpd[5171]: 88D3726A7D: client=localhost[127.0.0.1] Jul 9 07:52:54 ip-172-31-8-246 postfix/cleanup[5174]: 88D3726A7D: message-id=<1404892372.90900.YahooMailBasic@web163105.mail.bf1.yahoo.com> Jul 9 07:52:54 ip-172-31-8-246 postfix/qmgr[1119]: 88D3726A7D: from=<examplelist-bounces@example.com>, size=3672, nrcpt=1 (queue active) Jul 9 07:52:54 ip-172-31-8-246 postfix/smtpd[5171]: 95CF926A7E: client=localhost[127.0.0.1] Jul 9 07:52:54 ip-172-31-8-246 postfix/cleanup[5174]: 95CF926A7E: message-id=<1404892372.90900.YahooMailBasic@web163105.mail.bf1.yahoo.com> Jul 9 07:52:54 ip-172-31-8-246 postfix/smtpd[5171]: disconnect from localhost[127.0.0.1]
--/var/log/mailman/post----------------------------------------------- Jul 09 07:52:54 2014 (4216) post to examplelist from exampleuser@yahoo.com, size=3411, message-id=<1404892372.90900.YahooMailBasic@web163105.mail.bf1.yahoo.com>, success
-- Steven J. Owens stevenjowens@gmail.com puff@darksleep.com 412-401-8060
On 07/12/2014 10:58 AM, Steven Owens wrote:
What does mailman do to message IDs? When a message is posted to list subscribers, does it have the same message ID, or a predictably modified message ID that I can grep for?
Mailman does not modify the Message-ID of list posts except in the case of anonymous lists. In that case, and in the case of a message wrapped for DMARC mitigation, the Message-ID of the anonymized message or the outer wrapping message is a Mailman generated id of the form <mailman.nnnn.tttt.pppp.LISTNAME@HOST> where nnnn is a serial number, tttt is the time in seconds since the epoch, and pppp is the pid of the IncomingRunner process.
I'm trying to trace disappearing posts from a couple of list members. Their posts go through postfix, mailman reports the posts as succesfully posted, and the posts show up in the list archives, but not in the subscribers mailboxes.
Is there anything in Mailman's bounce log? What does Postfix report about the outgoing posts?
I have a small list with about 20 members. Most of the members, the list works fine; mail is showing up, being sent out, being received and replied to, etc.
A handful of members lots of bounces, probably due to their provider's policies (yahoo and hotmail, although *sometimes* the messages get through and one yahoo user has posted successfully).
However, for these two yahoo subscribers, their posts get reported as posted successfully, but are never received by the list subscribers.
Because the recipient's MTA's are rejecting the post for DMARC policy reasons.
I grep postfix logs (/var/log/mail.log) for the sender's email address, I see a message coming in, and postfix handing it off to mailman post (full log excerpt at end of message): ... Grepping for lines with a similar timestamp turns up half a dozen bounce messages two seconds later, which may explains why those 6 subscribers don't get the messages, but not the rest.
The bounces may be for multiple recipients in the same domain.
Jul 9 07:52:54 ip-172-31-8-246 postfix/smtpd[5171]: connect from localhost[127.0.0.1] Jul 9 07:52:54 ip-172-31-8-246 postfix/smtpd[5171]: 88D3726A7D: client=localhost[127.0.0.1] Jul 9 07:52:54 ip-172-31-8-246 postfix/cleanup[5174]: 88D3726A7D: message-id=<1404892372.90900.YahooMailBasic@web163105.mail.bf1.yahoo.com> Jul 9 07:52:54 ip-172-31-8-246 postfix/qmgr[1119]: 88D3726A7D: from=<examplelist-bounces@example.com>, size=3672, nrcpt=1 (queue active)
What else do you see in the Postfix log with queue ID 88D3726A7D ?
See the FAQ at <http://wiki.list.org/x/ggARAQ>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I have some new data just now, as one of the list subscribers finally reported that they had seen one of the posts in question.
Also, confusing the issue was the fact that my gmail account is receiving other list messages, but not list messages from those users, and there was no sign of a bounce from gmail. However, I did some googling and apparently it's not unknown for gmail to silently discard messages... sigh.
What's weird is that hotmail/yahoo/gmail are discarding or bouncing some messages, and allowing some others through. One yahoo user's posts get through while another's done.
Doing some more googling, it looks like a lot of spam filtering / blocking policies changed recently. I guess I need to study up. If you have any suggestions for where to start, I'm all ears (well, eyes).
I was going to ask if there's a log level setting that I can turn up, but google shows you replying to a similar request with "no" :-).
Thanks for the help.
-- Steven J. Owens stevenjowens@gmail.com puff@darksleep.com 412-401-8060
I'm not sure if this makes any difference but you'll want to make sure you have spf records & dkim set up for your mailserver. I had some troubles with messages being discarded by gmai, hotmail and others before I got this set up properly. On the other hand, if some messages are going through at all then I'm not sure if this would have much effect. In any event it's recommended.
On Sat, Jul 12, 2014 at 5:10 PM, Steven Owens <stevenjowens@gmail.com> wrote:
I have some new data just now, as one of the list subscribers finally reported that they had seen one of the posts in question.
Also, confusing the issue was the fact that my gmail account is receiving other list messages, but not list messages from those users, and there was no sign of a bounce from gmail. However, I did some googling and apparently it's not unknown for gmail to silently discard messages... sigh.
What's weird is that hotmail/yahoo/gmail are discarding or bouncing some messages, and allowing some others through. One yahoo user's posts get through while another's done.
Doing some more googling, it looks like a lot of spam filtering / blocking policies changed recently. I guess I need to study up. If you have any suggestions for where to start, I'm all ears (well, eyes).
I was going to ask if there's a log level setting that I can turn up, but google shows you replying to a similar request with "no" :-).
Thanks for the help.
-- Steven J. Owens stevenjowens@gmail.com puff@darksleep.com 412-401-8060
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/billynoah%40zuma-desig...
On 07/12/2014 02:28 PM, billy noah wrote:
I'm not sure if this makes any difference but you'll want to make sure you have spf records & dkim set up for your mailserver. I had some troubles with messages being discarded by gmai, hotmail and others before I got this set up properly. On the other hand, if some messages are going through at all then I'm not sure if this would have much effect. In any event it's recommended.
It is good practice and helps with delivery of some mail, but it has no effect on messages being rejected for DMARC policy reasons as the domain of your server's SPF and DKIM do not 'align' with the domain of the From: address.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Confusing things more is that not all Yahoo domains are set to p=reject. E.g. Yahoo.com.au doesn't, but yahoo.com does. Are all your yahoo members on yahoo.com?
Peter Shute
Sent from my iPad
On 13 Jul 2014, at 7:10 am, "Steven Owens" <stevenjowens@gmail.com> wrote:
I have some new data just now, as one of the list subscribers finally reported that they had seen one of the posts in question.
Also, confusing the issue was the fact that my gmail account is receiving other list messages, but not list messages from those users, and there was no sign of a bounce from gmail. However, I did some googling and apparently it's not unknown for gmail to silently discard messages... sigh.
What's weird is that hotmail/yahoo/gmail are discarding or bouncing some messages, and allowing some others through. One yahoo user's posts get through while another's done.
Doing some more googling, it looks like a lot of spam filtering / blocking policies changed recently. I guess I need to study up. If you have any suggestions for where to start, I'm all ears (well, eyes).
I was going to ask if there's a log level setting that I can turn up, but google shows you replying to a similar request with "no" :-).
Thanks for the help.
-- Steven J. Owens stevenjowens@gmail.com puff@darksleep.com 412-401-8060
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/pshute%40nuw.org.au
On 07/12/2014 02:10 PM, Steven Owens wrote:
Doing some more googling, it looks like a lot of spam filtering / blocking policies changed recently. I guess I need to study up. If you have any suggestions for where to start, I'm all ears (well, eyes).
There is an overwhelming amount of discussion in the archives of this list at <https://mail.python.org/pipermail/mailman-users/2014-April/thread.html> in threads with subjects mentioning DMARC and yahoo.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Steven Owens writes:
Also, confusing the issue was the fact that my gmail account is receiving other list messages, but not list messages from those users, and there was no sign of a bounce from gmail. However, I did some googling and apparently it's not unknown for gmail to silently discard messages... sigh.
I sympathize with your desire for more information, but it's not that simple.
A silent discard (or with report to DMARC Author Domain, not you or author, if requested by that domain) is correct and courteous behavior if Google can't confirm origin of your message.
Otherwise the bounces produce backscatter. Spammers are known to fake every header under the sun, but they can't fake SPF or DKIM.[1] It is not known precisely what Google's criteria are, but we do know that Google ignores DMARC p=reject for at least some mailing lists. However, I suspect that they want crypto proof of origin -- that means they can send the bounce to somebody who provably handled the message. Without DKIM or SPF, anywhere they send the bounce might be based on a spoofed address.
So the question posed elsewhere about whether your site provides DKIM and SPF, or not, becomes important. If not, you should do so as soon as possible -- that's the best way to improve reliability of delivery and receive bounces.
I was going to ask if there's a log level setting that I can turn up, but google shows you replying to a similar request with "no" :-).
What are you hoping to see logged? Maybe it could be added.
Footnotes: [1] Of course they sometimes can and do suborn the hosts that provide the signatures. I'm excluding that from "fake".
participants (5)
-
billy noah -
Mark Sapiro -
Peter Shute -
Stephen J. Turnbull -
Steven Owens