-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
I have Mailman up and running. I just sent an e-mail to 5 lists and the e-mail came duplicated to the recipients. I checked the header and the only difference is the following:
- Header:
X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1
- Header:
X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00 autolearn=ham version=3.3.1
How can a avoid duplicates in future? What is the reason?
BR Marco
On 03/18/2015 02:46 PM, Marco Stoecker wrote:
I have Mailman up and running. I just sent an e-mail to 5 lists and the e-mail came duplicated to the recipients. I checked the header and the only difference is the following:
- Header:
X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1
- Header:
X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00 autolearn=ham version=3.3.1
I'm confused. Are you looking at all the headers or just those your MUA shows you by default?
If you look at all the headers, at a minimum, some Received: headers and Mailman's X-BeenThere: header should be different.
How can a avoid duplicates in future? What is the reason?
Normally, if you cross post to 5 lists, people who are members of more than one of the lists will receive a copy of the post from each of the lists of which they are members.
The only way to avoid this is by using the Non-digest options -> regular_exclude_lists feature. For example, if you regularly post a single post to list1, list2, list3, list4 and list5, you could put list2, list3, list4 and list5 in list1's regular_exclude_lists, and put list3, list4 and list5 in list2's regular_exclude_lists, and put list4 and list5 in list3's regular_exclude_lists and finally put list5 in list4's regular_exclude_lists.
Then if a post is addressed to all 5 lists, anyone who is a member of list2, list3, list4 or list5 will not receive the post from list1 and anyone who is a member of list3, list4 or list5 will not receive the post from list2 and so on. See the (Details for regular_exclude_lists) link for more info.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/19/2015 05:22 PM, Mark Sapiro wrote:
On 03/18/2015 02:46 PM, Marco Stoecker wrote:
I have Mailman up and running. I just sent an e-mail to 5 lists and the e-mail came duplicated to the recipients. I checked the header and the only difference is the following:
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=unavailable version=3.3.1
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=ham version=3.3.1
I'm confused. Are you looking at all the headers or just those your MUA shows you by default?
Indeed, I looked only to the headers I recieved (Iceweasel->View->Message Source). Even now I checked those header my wife got, because she and I are in different lists, but received duplicates. Her headers seems identical.
If you look at all the headers, at a minimum, some Received: headers and Mailman's X-BeenThere: header should be different.
How can a avoid duplicates in future? What is the reason?
Normally, if you cross post to 5 lists, people who are members of more than one of the lists will receive a copy of the post from each of the lists of which they are members.
The only way to avoid this is by using the Non-digest options -> regular_exclude_lists feature. For example, if you regularly post a single post to list1, list2, list3, list4 and list5, you could put list2, list3, list4 and list5 in list1's regular_exclude_lists, and put list3, list4 and list5 in list2's regular_exclude_lists, and put list4 and list5 in list3's regular_exclude_lists and finally put list5 in list4's regular_exclude_lists.
Then if a post is addressed to all 5 lists, anyone who is a member of list2, list3, list4 or list5 will not receive the post from list1 and anyone who is a member of list3, list4 or list5 will not receive the post from list2 and so on. See the (Details for regular_exclude_lists) link for more info.
This hint is very useful to me as I have more lists with members that are in more than one list.
But in this dedicated case I wrote to 5 lists where the members are disjunct and my wife, and I got duplicates. I don't know why :-S
I assumed that the others got duplicates too, but I have to proof.
BR Marco
On 03/19/2015 11:31 AM, Marco Stoecker wrote:
On 03/19/2015 05:22 PM, Mark Sapiro wrote:
On 03/18/2015 02:46 PM, Marco Stoecker wrote:
I have Mailman up and running. I just sent an e-mail to 5 lists and the e-mail came duplicated to the recipients. I checked the header and the only difference is the following:
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=unavailable version=3.3.1
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=ham version=3.3.1
OK. So there was only one duplicate, yes? And that duplication occurred prior to the spam filtering that added the above header so presumably it wasn't caused by filtering rules in your MUA.
Indeed, I looked only to the headers I recieved (Iceweasel->View->Message Source). Even now I checked those header my wife got, because she and I are in different lists, but received duplicates. Her headers seems identical.
Are all the Received: headers identical including Queue IDs and time stamps? If so, the duplication must have occurred late in the delivery chain.
But in this dedicated case I wrote to 5 lists where the members are disjunct and my wife, and I got duplicates. I don't know why :-S
I assumed that the others got duplicates too, but I have to proof.
It difficult to say what's happening here without seeing the MTA logs from the various servers where duplication might have occurred, but unless there are entries in Mailman's smtp-failure log on the Mailman server, it is very unlikely that Mailman sent duplicates.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/19/2015 08:32 PM, Mark Sapiro wrote:
On 03/19/2015 11:31 AM, Marco Stoecker wrote:
On 03/19/2015 05:22 PM, Mark Sapiro wrote:
On 03/18/2015 02:46 PM, Marco Stoecker wrote:
I have Mailman up and running. I just sent an e-mail to 5 lists and the e-mail came duplicated to the recipients. I checked the header and the only difference is the following:
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=unavailable version=3.3.1
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=ham version=3.3.1
OK. So there was only one duplicate, yes? And that duplication occurred prior to the spam filtering that added the above header so presumably it wasn't caused by filtering rules in your MUA.
One duplicate I got and another one my wife got.
Indeed, I looked only to the headers I recieved (Iceweasel->View->Message Source). Even now I checked those header my wife got, because she and I are in different lists, but received duplicates. Her headers seems identical.
Are all the Received: headers identical including Queue IDs and time stamps? If so, the duplication must have occurred late in the delivery chain.
But in this dedicated case I wrote to 5 lists where the members are disjunct and my wife, and I got duplicates. I don't know why :-S
I assumed that the others got duplicates too, but I have to proof.
It difficult to say what's happening here without seeing the MTA logs from the various servers where duplication might have occurred, but unless there are entries in Mailman's smtp-failure log on the Mailman server, it is very unlikely that Mailman sent duplicates.
I got the section out of mail.log from the server during the time, it occured. But where do I find the smtp-failure log file?
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/marco%40stoecker-famil...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/19/2015 08:32 PM, Mark Sapiro wrote:
On 03/19/2015 11:31 AM, Marco Stoecker wrote:
On 03/19/2015 05:22 PM, Mark Sapiro wrote:
On 03/18/2015 02:46 PM, Marco Stoecker wrote:
I have Mailman up and running. I just sent an e-mail to 5 lists and the e-mail came duplicated to the recipients. I checked the header and the only difference is the following:
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=unavailable version=3.3.1
- Header: X-Spam-Status: No, score=-1.5 required=5.0
tests=BAYES_00 autolearn=ham version=3.3.1
OK. So there was only one duplicate, yes? And that duplication occurred prior to the spam filtering that added the above header so presumably it wasn't caused by filtering rules in your MUA.
Indeed, I looked only to the headers I recieved (Iceweasel->View->Message Source). Even now I checked those header my wife got, because she and I are in different lists, but received duplicates. Her headers seems identical.
Are all the Received: headers identical including Queue IDs and time stamps? If so, the duplication must have occurred late in the delivery chain.
But in this dedicated case I wrote to 5 lists where the members are disjunct and my wife, and I got duplicates. I don't know why :-S
I assumed that the others got duplicates too, but I have to proof.
It difficult to say what's happening here without seeing the MTA logs from the various servers where duplication might have occurred, but unless there are entries in Mailman's smtp-failure log on the Mailman server, it is very unlikely that Mailman sent duplicates.
Ahh, just found Mailman's log file folder: smtp-failure log is empty for yesterday but smtp log shows the following: Mar 18 22:17:05 2015 (2590) 5509EB1C.8050605@domain.de smtp to listA for 3 recips, completed in 13.036 seconds Mar 18 22:17:16 2015 (2590) 5509EB1C.8050605@domain.de smtp to listB for 14 recips, completed in 10.842 seconds Mar 18 22:17:17 2015 (2590) 5509EB1C.8050605@domain.de smtp to listC for 3 recips, completed in 1.194 seconds Mar 18 22:17:21 2015 (2590) 5509EB1C.8050605@domain.de smtp to listD for 13 recips, completed in 3.942 seconds Mar 18 22:17:24 2015 (2590) 5509EB1C.8050605@domain.de smtp to listE for 6 recips, completed in 2.504 seconds Mar 18 22:17:25 2015 (2590) 5509EB1C.8050605@domain.de smtp to listC for 3 recips, completed in 1.198 seconds Mar 18 22:17:29 2015 (2590) 5509EB1C.8050605@domain.de smtp to listB for 14 recips, completed in 3.784 seconds Mar 18 22:17:33 2015 (2590) 5509EB1C.8050605@domain.de smtp to listD for 13 recips, completed in 3.946 seconds Mar 18 22:17:35 2015 (2590) 5509EB1C.8050605@domain.de smtp to listE for 6 recips, completed in 2.515 seconds Mar 18 22:17:38 2015 (2590) 5509EB1C.8050605@domain.de smtp to listA for 3 recips, completed in 2.905 seconds
This ist the time I sent that e-mail to 5 lists (the announcement that the Mailman server is up and running and for future use) and it seems to me, that it was sent 2 times to each list, wasn't it?
On 3/19/2015 3:22 PM, Marco Stoecker wrote:
Ahh, just found Mailman's log file folder: smtp-failure log is empty for yesterday but smtp log shows the following: Mar 18 22:17:05 2015 (2590) 5509EB1C.8050605@domain.de smtp to listA for 3 recips, completed in 13.036 seconds Mar 18 22:17:16 2015 (2590) 5509EB1C.8050605@domain.de smtp to listB for 14 recips, completed in 10.842 seconds Mar 18 22:17:17 2015 (2590) 5509EB1C.8050605@domain.de smtp to listC for 3 recips, completed in 1.194 seconds Mar 18 22:17:21 2015 (2590) 5509EB1C.8050605@domain.de smtp to listD for 13 recips, completed in 3.942 seconds Mar 18 22:17:24 2015 (2590) 5509EB1C.8050605@domain.de smtp to listE for 6 recips, completed in 2.504 seconds Mar 18 22:17:25 2015 (2590) 5509EB1C.8050605@domain.de smtp to listC for 3 recips, completed in 1.198 seconds Mar 18 22:17:29 2015 (2590) 5509EB1C.8050605@domain.de smtp to listB for 14 recips, completed in 3.784 seconds Mar 18 22:17:33 2015 (2590) 5509EB1C.8050605@domain.de smtp to listD for 13 recips, completed in 3.946 seconds Mar 18 22:17:35 2015 (2590) 5509EB1C.8050605@domain.de smtp to listE for 6 recips, completed in 2.515 seconds Mar 18 22:17:38 2015 (2590) 5509EB1C.8050605@domain.de smtp to listA for 3 recips, completed in 2.905 seconds
This ist the time I sent that e-mail to 5 lists (the announcement that the Mailman server is up and running and for future use) and it seems to me, that it was sent 2 times to each list, wasn't it?
Yes, but if the post was sent twice to each list, the Received: headers of the two posts you received must differ in detail.
Anyway, look at the MTA log and see if there were two deliveries of the post to each Mailman list at around 22:17.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/19/2015 11:55 PM, Mark Sapiro wrote:
Yes, but if the post was sent twice to each list, the Received: headers of the two posts you received must differ in detail.
I copied the "View->Message Source" information in two text files and compared them with diff, but no difference. What does this mean?
Anyway, look at the MTA log and see if there were two deliveries of the post to each Mailman list at around 22:17.
Thx Mark
Here is the logfile mail.info (I guess its written by postfix)
Mar 18 22:16:47 hostname postfix/local[9821]: 6D42445B: to=listA@localhost, relay=local, delay=4, delays=2.8/0.13/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA) Mar 18 22:16:47 hostname postfix/local[9825]: 6D42445B: to=listB@localhost, relay=local, delay=4, delays=2.8/0.19/0/0.93, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listB) Mar 18 22:16:47 hostname postfix/local[9827]: 6D42445B: to=listC@localhost, relay=local, delay=4, delays=2.8/0.33/0/0.81, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listC) Mar 18 22:16:47 hostname postfix/local[9820]: 6D42445B: to=listD@localhost, relay=local, delay=4, delays=2.8/0.11/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listD) Mar 18 22:16:47 hostname postfix/local[9823]: 6D42445B: to=listE@localhost, relay=local, delay=4, delays=2.8/0.17/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listE)
... and a bit later
Mar 18 22:16:49 hostname postfix/local[9827]: 4ACCAD07: to=listB@localhost, relay=local, delay=2.9, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listB) Mar 18 22:16:49 hostname postfix/local[9825]: 4ACCAD07: to=listC@localhost, relay=local, delay=2.9, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listC) Mar 18 22:16:49 hostname postfix/local[9821]: 4ACCAD07: to=listD@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listD) Mar 18 22:16:49 hostname postfix/local[9820]: 4ACCAD07: to=listE@localhost, relay=local, delay=3, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listE) Mar 18 22:16:49 hostname postfix/local[9823]: 4ACCAD07: to=listA@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA)
So what is it, a Postfix or a Mailman issue?
BR Marco
On 03/20/2015 07:32 AM, Marco Stoecker wrote:
On 03/19/2015 11:55 PM, Mark Sapiro wrote:
Yes, but if the post was sent twice to each list, the Received: headers of the two posts you received must differ in detail.
I copied the "View->Message Source" information in two text files and compared them with diff, but no difference. What does this mean?
This is very difficult to understand. At the very least, Postfix's Received: headers should include time stamps and queue IDs which would be different for the two messages.
Here is the logfile mail.info (I guess its written by postfix)
Mar 18 22:16:47 hostname postfix/local[9821]: 6D42445B: to=listA@localhost, relay=local, delay=4, delays=2.8/0.13/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA)
...
Mar 18 22:16:49 hostname postfix/local[9823]: 4ACCAD07: to=listA@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA)
So what is it, a Postfix or a Mailman issue?
It is somewhere between your mail client and Postfix on the Mailman server. It's not Mailman. Are you certain you didn't just send the message twice?
What are the rest of the postfix log messages from 'grep 6D42445B' and 'grep 4ACCAD07' of the log file?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
It is somewhere between your mail client and Postfix on the Mailman server. It's not Mailman. Are you certain you didn't just send the message twice?
I'm very sure because in the outbox it appears only ones. And I copied the headers again, just to make sure I do not made a mistake the first time. But again no difference with commandline comannd 'diff'. That is very curious :-(
What are the rest of the postfix log messages from 'grep 6D42445B' and 'grep 4ACCAD07' of the log file?
and here it comes...
for grep 6D42445B:
Mar 18 22:16:43 hostname postfix/smtpd[9815]: 6D42445B: client=localhost[::1] Mar 18 22:16:43 hostname postfix/cleanup[9818]: 6D42445B: message-id=5509EB1C.8050605@stoecker-family.de Mar 18 22:16:46 hostname postfix/qmgr[3081]: 6D42445B: from=marco@stoecker-family.de, size=6653, nrcpt=5 (queue active) Mar 18 22:16:47 hostname postfix/local[9821]: 6D42445B: to=listA@localhost, relay=local, delay=4, delays=2.8/0.13/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA) Mar 18 22:16:47 hostname postfix/local[9825]: 6D42445B: to=listB@localhost, relay=local, delay=4, delays=2.8/0.19/0/0.93, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listB) Mar 18 22:16:47 hostname postfix/local[9827]: 6D42445B: to=listC@localhost, relay=local, delay=4, delays=2.8/0.33/0/0.81, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listC) Mar 18 22:16:47 hostname postfix/local[9820]: 6D42445B: to=listD@localhost, relay=local, delay=4, delays=2.8/0.11/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listD) Mar 18 22:16:47 hostname postfix/local[9823]: 6D42445B: to=listE@localhost, relay=local, delay=4, delays=2.8/0.17/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listE) Mar 18 22:16:47 hostname postfix/qmgr[3081]: 6D42445B: removed
and for grep 4ACCAD07:
Mar 18 22:16:46 hostname postfix/smtpd[9815]: 4ACCAD07: client=localhost[::1] Mar 18 22:16:46 hostname postfix/cleanup[9818]: 4ACCAD07: message-id=5509EB1C.8050605@stoecker-family.de Mar 18 22:16:48 hostname postfix/qmgr[3081]: 4ACCAD07: from=marco@stoecker-family.de, size=6652, nrcpt=5 (queue active) Mar 18 22:16:49 hostname postfix/local[9827]: 4ACCAD07: to=listB@localhost, relay=local, delay=2.9, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listB) Mar 18 22:16:49 hostname postfix/local[9825]: 4ACCAD07: to=listC@localhost, relay=local, delay=2.9, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listC) Mar 18 22:16:49 hostname postfix/local[9821]: 4ACCAD07: to=listD@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listD) Mar 18 22:16:49 hostname postfix/local[9820]: 4ACCAD07: to=listE@localhost, relay=local, delay=3, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listE) Mar 18 22:16:49 hostname postfix/local[9823]: 4ACCAD07: to=listA@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA) Mar 18 22:16:49 hostname postfix/qmgr[3081]: 4ACCAD07: removed
BR Marco
On 03/20/2015 08:42 AM, Marco Stoecker wrote:
What are the rest of the postfix log messages from 'grep 6D42445B' and 'grep 4ACCAD07' of the log file?
and here it comes...
for grep 6D42445B:
Mar 18 22:16:43 hostname postfix/smtpd[9815]: 6D42445B: client=localhost[::1] Mar 18 22:16:43 hostname postfix/cleanup[9818]: 6D42445B: message-id=5509EB1C.8050605@stoecker-family.de
OK. This message was received from something on localhost (IPv6 [::1] at 22:16:43 and delivered to all the Mailman list recipients.
Mar 18 22:16:46 hostname postfix/qmgr[3081]: 6D42445B: from=marco@stoecker-family.de, size=6653, nrcpt=5 (queue active) Mar 18 22:16:47 hostname postfix/local[9821]: 6D42445B: to=listA@localhost, relay=local, delay=4, delays=2.8/0.13/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA) Mar 18 22:16:47 hostname postfix/local[9825]: 6D42445B: to=listB@localhost, relay=local, delay=4, delays=2.8/0.19/0/0.93, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listB) Mar 18 22:16:47 hostname postfix/local[9827]: 6D42445B: to=listC@localhost, relay=local, delay=4, delays=2.8/0.33/0/0.81, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listC) Mar 18 22:16:47 hostname postfix/local[9820]: 6D42445B: to=listD@localhost, relay=local, delay=4, delays=2.8/0.11/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listD) Mar 18 22:16:47 hostname postfix/local[9823]: 6D42445B: to=listE@localhost, relay=local, delay=4, delays=2.8/0.17/0/0.98, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listE) Mar 18 22:16:47 hostname postfix/qmgr[3081]: 6D42445B: removed
and for grep 4ACCAD07:
Mar 18 22:16:46 hostname postfix/smtpd[9815]: 4ACCAD07: client=localhost[::1] Mar 18 22:16:46 hostname postfix/cleanup[9818]: 4ACCAD07: message-id=5509EB1C.8050605@stoecker-family.de
And then 3 seconds later something on the same host delivered a second copy to Postfix which delivered that to the lists also.
Mar 18 22:16:48 hostname postfix/qmgr[3081]: 4ACCAD07: from=marco@stoecker-family.de, size=6652, nrcpt=5 (queue active) Mar 18 22:16:49 hostname postfix/local[9827]: 4ACCAD07: to=listB@localhost, relay=local, delay=2.9, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listB) Mar 18 22:16:49 hostname postfix/local[9825]: 4ACCAD07: to=listC@localhost, relay=local, delay=2.9, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listC) Mar 18 22:16:49 hostname postfix/local[9821]: 4ACCAD07: to=listD@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listD) Mar 18 22:16:49 hostname postfix/local[9820]: 4ACCAD07: to=listE@localhost, relay=local, delay=3, delays=1.9/0.01/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listE) Mar 18 22:16:49 hostname postfix/local[9823]: 4ACCAD07: to=listA@localhost, relay=local, delay=3, delays=1.9/0/0/1, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post listA) Mar 18 22:16:49 hostname postfix/qmgr[3081]: 4ACCAD07: removed
So this duplication occurred before Mailman and even before Postfix delivering to Mailman. It could be anything. You could try grepping the log for the Message-ID and see if that gives a clue.
Look for an original arrival being passed to some process that returns two copies.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/20/2015 05:12 PM, Mark Sapiro wrote:
On 03/20/2015 08:42 AM, Marco Stoecker wrote:
Mar 18 22:16:43 hostname postfix/smtpd[9815]: 6D42445B: client=localhost[::1] Mar 18 22:16:43 hostname postfix/cleanup[9818]: 6D42445B: message-id=5509EB1C.8050605@stoecker-family.de
OK. This message was received from something on localhost (IPv6 [::1] at 22:16:43 and delivered to all the Mailman list recipients.
[snipped]
Mar 18 22:16:47 hostname postfix/qmgr[3081]: 6D42445B: removed
and for grep 4ACCAD07:
Mar 18 22:16:46 hostname postfix/smtpd[9815]: 4ACCAD07: client=localhost[::1] Mar 18 22:16:46 hostname postfix/cleanup[9818]: 4ACCAD07: message-id=5509EB1C.8050605@stoecker-family.de
And then 3 seconds later something on the same host delivered a second copy to Postfix which delivered that to the lists also.
[snipped]
So this duplication occurred before Mailman and even before Postfix delivering to Mailman. It could be anything. You could try grepping the log for the Message-ID and see if that gives a clue.
Look for an original arrival being passed to some process that returns two copies.
I also use clamav-milter and spamassassin-milter together with postfix, maybe ones of these is the root cause. Thx for your detailed help. I think for Mailman this is solved, cause it has nothing to do with it.
BR Marco
On 03/20/2015 05:29 PM, Marco Stoecker wrote:
On 03/20/2015 05:12 PM, Mark Sapiro wrote:
On 03/20/2015 08:42 AM, Marco Stoecker wrote:
Mar 18 22:16:43 hostname postfix/smtpd[9815]: 6D42445B: client=localhost[::1] Mar 18 22:16:43 hostname postfix/cleanup[9818]: 6D42445B: message-id=5509EB1C.8050605@stoecker-family.de
OK. This message was received from something on localhost (IPv6 [::1] at 22:16:43 and delivered to all the Mailman list recipients.
[snipped]
Mar 18 22:16:47 hostname postfix/qmgr[3081]: 6D42445B: removed
and for grep 4ACCAD07:
Mar 18 22:16:46 hostname postfix/smtpd[9815]: 4ACCAD07: client=localhost[::1] Mar 18 22:16:46 hostname postfix/cleanup[9818]: 4ACCAD07: message-id=5509EB1C.8050605@stoecker-family.de
And then 3 seconds later something on the same host delivered a second copy to Postfix which delivered that to the lists also.
[snipped]
So this duplication occurred before Mailman and even before Postfix delivering to Mailman. It could be anything. You could try grepping the log for the Message-ID and see if that gives a clue.
Look for an original arrival being passed to some process that returns two copies.
I also use clamav-milter and spamassassin-milter together with postfix, maybe ones of these is the root cause. Thx for your detailed help. I think for Mailman this is solved, cause it has nothing to do with it.
I see a pattern now, when duplicates occur and it may have to do with mailman. If someone in a list send an email to two or three lists on the same server, the recipients of the list will get two or three times the same email. If it is sent to one list only, no duplicate occur. Any glue what the root cause could be?
BR Marco
On 10/20/2015 04:45 PM, Marco Stoecker wrote:
I see a pattern now, when duplicates occur and it may have to do with mailman. If someone in a list send an email to two or three lists on the same server, the recipients of the list will get two or three times the same email. If it is sent to one list only, no duplicate occur. Any glue what the root cause could be?
If you are saying that people who are members of more than one list receive a copy from each list of which they are a member when a post is sent to multiple lists, that's the way Mailman works.
I.e. If I am a member of list1, list2 and list3 and someone posts to all three lists, I will receive a copy from each list.
The list member setting "Avoid duplicate copies of messages?" (nodupes) does not affect this. It only affects whether I receive a list copy if I am also a To: or Cc: addressee of the post.
The list's Non-digest options -> regular_exclude_lists can modify this behavior somewhat. E.g., if the regular_exclude_lists setting for list1@example.com includes list2@example.com and list3@example.com and the regular_exclude_lists setting for list2@example.com includes list3@example.com, then a member of all three lists will receive a copy of a post sent to all three lists from only list3@example.com.
regular_exclude_lists need to be set up with care. In particular, if list2@example.com is in list1@example.com's regular_exclude_lists, list1@example.com MUST not be in list2@example.com's regular_exclude_lists or a post sent to both lists will not be received by anyone who is a member of both lists.
Note that in the above, member means non-digest member. Digest members are not affected by this.
On 10/21/2015 02:40 AM, Mark Sapiro wrote:
On 10/20/2015 04:45 PM, Marco Stoecker wrote:
I see a pattern now, when duplicates occur and it may have to do with mailman. If someone in a list send an email to two or three lists on the same server, the recipients of the list will get two or three times the same email. If it is sent to one list only, no duplicate occur. Any glue what the root cause could be?
If you are saying that people who are members of more than one list receive a copy from each list of which they are a member when a post is sent to multiple lists, that's the way Mailman works.
I.e. If I am a member of list1, list2 and list3 and someone posts to all three lists, I will receive a copy from each list.
No, what I was trying to say is, I have a list member which is also allowed (via accept_these_nonmembers) to write to other lists but is not a member of these other lists. So if x@y.com sent an email to list1@mydomain.com and to list2@mydomain.com and x@y.com is only member of list1@mydomain.com than x@y.com gets a duplicate email as well as all members of these two lists. Furthermore if that email is also sent to list3@mydomain.com, than the members as well as x@y.com will receive 3 identical emails (content wise). It seems to me as if mailman, when it hands over to postfix, duplicates the email so many times as to so many lists the emails was sent. But this is only my suggestion. I have no idea, where to look into. The mailman logs and postfix logs gave me no clear picture.
The list member setting "Avoid duplicate copies of messages?" (nodupes) does not affect this. It only affects whether I receive a list copy if I am also a To: or Cc: addressee of the post.
This is already set in the intended manner.
The list's Non-digest options -> regular_exclude_lists can modify this behavior somewhat. E.g., if the regular_exclude_lists setting for list1@example.com includes list2@example.com and list3@example.com and the regular_exclude_lists setting for list2@example.com includes list3@example.com, then a member of all three lists will receive a copy of a post sent to all three lists from only list3@example.com.
I do not use the regular_exclude_lists
regular_exclude_lists need to be set up with care. In particular, if list2@example.com is in list1@example.com's regular_exclude_lists, list1@example.com MUST not be in list2@example.com's regular_exclude_lists or a post sent to both lists will not be received by anyone who is a member of both lists.
Note that in the above, member means non-digest member. Digest members are not affected by this.
On 10/20/2015 11:15 PM, Marco Stoecker wrote:
No, what I was trying to say is, I have a list member which is also allowed (via accept_these_nonmembers) to write to other lists but is not a member of these other lists. So if x@y.com sent an email to list1@mydomain.com and to list2@mydomain.com and x@y.com is only member of list1@mydomain.com than x@y.com gets a duplicate email as well as all members of these two lists.
I'm still confused. Are you saying here that someone who is a member of say list1 and not list2 will get two copies of the message?. If so, look at the complete headers of both messages. Do they both come from Mailman and does one of them come from list1 and one from list2?
Furthermore if that email is also sent to list3@mydomain.com, than the members as well as x@y.com will receive 3 identical emails (content wise). It seems to me as if mailman, when it hands over to postfix, duplicates the email so many times as to so many lists the emails was sent. But this is only my suggestion. I have no idea, where to look into. The mailman logs and postfix logs gave me no clear picture.
The Mailman 'smtp' log will have entries like
Oct 18 17:47:02 2015 (pppp) <message-id> smtp to listname for nnn recips, completed in t.ttt seconds
where pppp is the PID of OutgoingRunner, message-id is the actual message-id of the post, listname is the list name and nnn is the number of recipients sent to.
Postfix's log will have complete information as to what was delivered to Mailman and received from Mailman and subsequently delivered.
Mailman's logs may be in /var/lib/mailman/logs/, /usr/local/mailman/logs, /var/log/mailman/ or ??? depending on how mailman was installed. Postfix's logs are usually /var/log/maillog or /var/log/mail.log.
On 10/21/2015 05:56 PM, Mark Sapiro wrote:
On 10/20/2015 11:15 PM, Marco Stoecker wrote:
No, what I was trying to say is, I have a list member which is also allowed (via accept_these_nonmembers) to write to other lists but is not a member of these other lists. So if x@y.com sent an email to list1@mydomain.com and to list2@mydomain.com and x@y.com is only member of list1@mydomain.com than x@y.com gets a duplicate email as well as all members of these two lists.
I'm still confused. Are you saying here that someone who is a member of say list1 and not list2 will get two copies of the message?. If so, look at the complete headers of both messages. Do they both come from Mailman and does one of them come from list1 and one from list2?
That is exactly what I've tried to say :-) And not only that 'someone' but also all members of list1 and list2 get two copies (ore more).
How do I identify whether the message comes from mailman or list1 or list2? Sorry for the questions, I'm not that experienced so far.
Furthermore if that email is also sent to list3@mydomain.com, than the members as well as x@y.com will receive 3 identical emails (content wise). It seems to me as if mailman, when it hands over to postfix, duplicates the email so many times as to so many lists the emails was sent. But this is only my suggestion. I have no idea, where to look into. The mailman logs and postfix logs gave me no clear picture.
The Mailman 'smtp' log will have entries like
Will have I look into it and come back here
Oct 18 17:47:02 2015 (pppp) <message-id> smtp to listname for nnn recips, completed in t.ttt seconds
where pppp is the PID of OutgoingRunner, message-id is the actual message-id of the post, listname is the list name and nnn is the number of recipients sent to.
Postfix's log will have complete information as to what was delivered to Mailman and received from Mailman and subsequently delivered.
Mailman's logs may be in /var/lib/mailman/logs/, /usr/local/mailman/logs, /var/log/mailman/ or ??? depending on how mailman was installed. Postfix's logs are usually /var/log/maillog or /var/log/mail.log.
On 10/21/2015 02:14 PM, Marco Stoecker wrote:
How do I identify whether the message comes from mailman or list1 or list2? Sorry for the questions, I'm not that experienced so far.
Here are some of the headers of your message as I received it from the mailman-users list with my comments interspersed. How you actually see these headers depends on your mail client. It may be something like "show original" or "view source" or something else depending on the client. I have removed some headers which aren't relevant to this discussion.
Return-Path: mailman-users-bounces+mark=msapiro.net@python.org
Above shows the envelope sender of the message which is the VERPed mailman-users-bounces address.
X-Original-To: mark@msapiro.net Delivered-To: msapiro_mark@sbh16.songbird.com
Above are from my Postfix.
Received: from mail.python.org (mail.python.org [82.94.164.166]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sbh16.songbird.com (Postfix) with ESMTPS id 4CDC511E0176 for mark@msapiro.net; Wed, 21 Oct 2015 14:14:41 -0700 (PDT)
From my Postfix showing receipt from mail.python.org and id 4CDC511E0176
which I can find in my Postfix logs.
Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3nh4Mg1yt4zR5g for mark@msapiro.net; Wed, 21 Oct 2015 23:14:39 +0200 (CEST)
Postfix on albatross.python.org (another name for mail.python.org) received the message from Mailman on mail.python.org.
X-Original-To: mailman-users@python.org Delivered-To: mailman-users@mail.python.org Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3nh4M35Td4zQHp for mailman-users@python.org; Wed, 21 Oct 2015 23:14:07 +0200 (CEST)
mail.python.org receiving and delivering the message to Mailman
(spam_bayes related headers removed)
Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 21 Oct 2015 23:14:07 +0200
Message passing to spam_bayes processing on albatross.python.org.
Received: from metaxa.loswebos.de (metaxa.loswebos.de [213.187.93.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.python.org (Postfix) with ESMTPS for mailman-users@python.org; Wed, 21 Oct 2015 23:14:06 +0200 (CEST)
Message received from your relay metaxa.loswebos.de by mail.python.org.
Received: from [192.168.178.20] (ipbcc26d30.dynamic.kabel-deutschland.de [188.194.109.48]) by metaxa.loswebos.de (Postfix) with ESMTPSA id D09F03A8C012 for mailman-users@python.org; Wed, 21 Oct 2015 23:14:04 +0200 (CEST)
Message received from you by your relay.
Note that all the above headers are added by MTAs to the beginning of the message so the "higher up" they are, the more recent they are.
Message-ID: 5628001C.6020400@stoecker-family.de Date: Wed, 21 Oct 2015 23:14:04 +0200 From: Marco Stoecker marco@stoecker-family.de User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.8.0 MIME-Version: 1.0 To: mailman-users@python.org References: 5509F231.4010102@stoecker-family.de 550AF7BE.4010208@msapiro.net 550B15EC.20603@stoecker-family.de 550B2460.8020100@msapiro.net 550B4C0D.2000302@stoecker-family.de 550B53E6.5020209@msapiro.net 550C2F73.7050800@stoecker-family.de 550C34ED.2070207@msapiro.net 550C3FF6.6010306@stoecker-family.de 550C46EC.6020900@msapiro.net 550C4ADC.9050303@stoecker-family.de 5626D222.9070103@stoecker-family.de 5626DEF9.3090400@msapiro.net 56272D7A.20603@stoecker-family.de 5627B5BA.2000702@msapiro.net In-Reply-To: 5627B5BA.2000702@msapiro.net
The group above are all in your original message as sent.
Subject: Re: [Mailman-Users] duplicates
This may have been modified by Mailman's subject prefixing.
X-BeenThere: mailman-users@python.org
This was added by mailman to say the message has been processed by the mailman-users@python.org list.
X-Mailman-Version: 2.1.20+ Precedence: list List-Id: Mailman mailing list management users <mailman-users.python.org> List-Unsubscribe: https://mail.python.org/mailman/options/mailman-users, mailto:mailman-users-request@python.org?subject=unsubscribe List-Archive: http://mail.python.org/pipermail/mailman-users/ List-Post: mailto:mailman-users@python.org List-Help: mailto:mailman-users-request@python.org?subject=help List-Subscribe: https://mail.python.org/mailman/listinfo/mailman-users, mailto:mailman-users-request@python.org?subject=subscribe
Added by Mailman.
Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit
Possibly modified by Mailman's content filtering but not in this case since there is no X-Content-Filtered-By: header.
Errors-To: mailman-users-bounces+mark=msapiro.net@python.org Sender: "Mailman-Users" mailman-users-bounces+mark=msapiro.net@python.org
Added by Mailman.
If you just get the complete headers from all the duplicates of one message received by one user and post those, I can help interpret them.
On 10/21/2015 11:45 PM, Mark Sapiro wrote:
On 10/21/2015 02:14 PM, Marco Stoecker wrote:
How do I identify whether the message comes from mailman or list1 or list2? Sorry for the questions, I'm not that experienced so far.
Here are some of the headers of your message as I received it from the mailman-users list with my comments interspersed. How you actually see these headers depends on your mail client. It may be something like "show original" or "view source" or something else depending on the client. I have removed some headers which aren't relevant to this discussion.
Return-Path: mailman-users-bounces+mark=msapiro.net@python.org
Above shows the envelope sender of the message which is the VERPed mailman-users-bounces address.
<snipped>
Added by Mailman.
If you just get the complete headers from all the duplicates of one message received by one user and post those, I can help interpret them.
This would be very great! I did attach the header from a recent message I got last week. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
But the email, which I sent last week, with the attachement never made it to this list. How can I send the headers instead, cause this email would be very long?
Did I understood right, all entry in /var/log/mailman/smtp file are made by mailman and in this case it could be an indicator that mailman send the email 5 times to each list?
And here is the corresponding entry in the /var/log/mailman/smtp file:
Oct 22 10:40:33 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 21.833 seconds Oct 22 10:41:02 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 28.426 seconds Oct 22 10:41:12 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.447 seconds Oct 22 10:41:23 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 11.236 seconds Oct 22 10:41:29 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 5.095 seconds Oct 22 10:41:39 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.615 seconds Oct 22 10:41:50 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.871 seconds Oct 22 10:41:55 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 4.794 seconds Oct 22 10:42:06 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.890 seconds Oct 22 10:42:22 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 15.688 seconds Oct 22 10:42:32 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.289 seconds Oct 22 10:42:43 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.736 seconds Oct 22 10:42:53 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.497 seconds Oct 22 10:43:09 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 16.035 seconds Oct 22 10:43:15 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 5.468 seconds Oct 22 10:43:25 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.680 seconds Oct 22 10:43:31 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 5.253 seconds Oct 22 10:43:46 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 15.867 seconds Oct 22 10:43:57 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.956 seconds Oct 22 10:44:09 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 11.101 seconds Oct 22 10:44:18 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 9.922 seconds Oct 22 10:44:28 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 9.971 seconds Oct 22 10:44:39 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.315 seconds Oct 22 10:44:55 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 16.343 seconds Oct 22 10:45:00 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 4.947 seconds
BR Marco
On 10/26/2015 02:14 PM, Marco Stoecker wrote:
I did attach the header from a recent message I got last week. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
But the email, which I sent last week, with the attachement never made it to this list. How can I send the headers instead, cause this email would be very long?
I never saw such a message, even in the moderation queue because of size.
You can view the raw message, select all the headers, copy and paste them into a post.
Did I understood right, all entry in /var/log/mailman/smtp file are made by mailman and in this case it could be an indicator that mailman send the email 5 times to each list?
Yes, the smtp log entries indicate each of the 5 lists received 5 messages with message-id 008b01d10ca4$eb5fd0b0$c21f7210$@de.
Also, delivery from Mailman to Postfix seems slow, but that's a separate issue and may not be a problem for you.
And here is the corresponding entry in the /var/log/mailman/smtp file:
Oct 22 10:40:33 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 21.833 seconds
...
Oct 22 10:41:39 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.615 seconds
...
Oct 22 10:42:32 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.289 seconds
...
Oct 22 10:44:09 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 11.101 seconds
...
Oct 22 10:44:28 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 9.971 seconds
So we know that each list is receiving 5 copies of the message. I'm am certain this duplication occurs in the MTA delivering to Mailman or before and not in Mailman itself.
The most helpful information would be the Postfix log entries from several minutes before Oct 22 10:40:33 2015 up to Oct 22 10:44:28 2015,
On 10/26/2015 11:32 PM, Mark Sapiro wrote:
On 10/26/2015 02:14 PM, Marco Stoecker wrote:
I did attach the header from a recent message I got last week. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
But the email, which I sent last week, with the attachement never made it to this list. How can I send the headers instead, cause this email would be very long?
I never saw such a message, even in the moderation queue because of size.
You can view the raw message, select all the headers, copy and paste them into a post.
Did I understood right, all entry in /var/log/mailman/smtp file are made by mailman and in this case it could be an indicator that mailman send the email 5 times to each list?
Yes, the smtp log entries indicate each of the 5 lists received 5 messages with message-id 008b01d10ca4$eb5fd0b0$c21f7210$@de.
Also, delivery from Mailman to Postfix seems slow, but that's a separate issue and may not be a problem for you.
And here is the corresponding entry in the /var/log/mailman/smtp file:
Oct 22 10:40:33 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 21.833 seconds
...
Oct 22 10:41:39 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.615 seconds
...
Oct 22 10:42:32 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.289 seconds
...
Oct 22 10:44:09 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 11.101 seconds
...
Oct 22 10:44:28 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 9.971 seconds
So we know that each list is receiving 5 copies of the message. I'm am certain this duplication occurs in the MTA delivering to Mailman or before and not in Mailman itself.
The most helpful information would be the Postfix log entries from several minutes before Oct 22 10:40:33 2015 up to Oct 22 10:44:28 2015,
This is the related mail.log entry:
Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: connect from localhost[::1] Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: BF9AD1C94: client=localhost[::1] Oct 22 10:40:05 wakis02 postfix/cleanup[4901]: BF9AD1C94: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:10 wakis02 postfix/qmgr[1304]: BF9AD1C94: from=alexandrakick@web.de, size=312694, nrcpt=5 (queue active) Oct 22 10:40:10 wakis02 postfix/smtpd[4898]: ADDE72652: client=localhost[::1] Oct 22 10:40:10 wakis02 postfix/cleanup[4901]: ADDE72652: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:10 wakis02 postfix/local[4903]: BF9AD1C94: to=beisitz@localhost, relay=local, delay=5.2, delays=4.9/0.03/0/0.3, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post beisitz) Oct 22 10:40:10 wakis02 postfix/local[4909]: BF9AD1C94: to=vorstand@localhost, relay=local, delay=5.3, delays=4.9/0.11/0/0.29, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post vorstand) Oct 22 10:40:10 wakis02 postfix/local[4907]: BF9AD1C94: to=kollegium@localhost, relay=local, delay=5.3, delays=4.9/0.07/0/0.33, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post kollegium) Oct 22 10:40:10 wakis02 postfix/local[4905]: BF9AD1C94: to=gruppensprecher@localhost, relay=local, delay=5.3, delays=4.9/0.05/0/0.36, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post gruppensprecher) Oct 22 10:40:10 wakis02 postfix/local[4902]: BF9AD1C94: to=ak-leiter@localhost, relay=local, delay=5.3, delays=4.9/0.02/0/0.39, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post ak-leiter) Oct 22 10:40:11 wakis02 postfix/qmgr[1304]: BF9AD1C94: removed Oct 22 10:40:12 wakis02 postfix/smtpd[4920]: connect from localhost[::1] Oct 22 10:40:12 wakis02 postfix/smtpd[4920]: 2AFF52697: client=localhost[::1] Oct 22 10:40:12 wakis02 postfix/cleanup[4921]: 2AFF52697: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:21 wakis02 postfix/qmgr[1304]: ADDE72652: from=alexandrakick@web.de, size=312704, nrcpt=5 (queue active) Oct 22 10:40:21 wakis02 postfix/local[4909]: ADDE72652: to=vorstand@localhost, relay=local, delay=11, delays=11/0.01/0/0.35, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post vorstand) Oct 22 10:40:21 wakis02 postfix/local[4905]: ADDE72652: to=gruppensprecher@localhost, relay=local, delay=11, delays=11/0/0/0.35, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post gruppensprecher) Oct 22 10:40:21 wakis02 postfix/local[4903]: ADDE72652: to=ak-leiter@localhost, relay=local, delay=11, delays=11/0/0/0.35, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post ak-leiter) Oct 22 10:40:21 wakis02 postfix/local[4902]: ADDE72652: to=beisitz@localhost, relay=local, delay=11, delays=11/0/0/0.36, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post beisitz) Oct 22 10:40:21 wakis02 postfix/local[4907]: ADDE72652: to=kollegium@localhost, relay=local, delay=11, delays=11/0.01/0/0.36, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post kollegium) Oct 22 10:40:21 wakis02 postfix/smtpd[4898]: 8010E3A: client=localhost[::1] Oct 22 10:40:21 wakis02 postfix/cleanup[4901]: 8010E3A: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:21 wakis02 postfix/qmgr[1304]: ADDE72652: removed Oct 22 10:40:23 wakis02 postfix/qmgr[1304]: 2AFF52697: from=beisitz-bounces@waldorfkindergarten-erlangen.de, size=314124, nrcpt=1 (queue active) Oct 22 10:40:23 wakis02 postfix/smtpd[4920]: 63AC226AE: client=localhost[::1] Oct 22 10:40:23 wakis02 postfix/cleanup[4921]: 63AC226AE: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:27 wakis02 postfix/smtp[4937]: 2AFF52697: to=N.Rausch@gmx.net, relay=smtp.1und1.de[212.227.15.183]:25, delay=16, delays=11/0.17/0.17/4.1, dsn=2.0.0, status=sent (250 Requested mail action okay, completed: id=0MUQrq-1ZyZK12VWs-00R3sy) Oct 22 10:40:27 wakis02 postfix/qmgr[1304]: 2AFF52697: removed Oct 22 10:40:32 wakis02 postfix/qmgr[1304]: 8010E3A: from=alexandrakick@web.de, size=312694, nrcpt=5 (queue active) Oct 22 10:40:32 wakis02 postfix/local[4903]: 8010E3A: to=gruppensprecher@localhost, relay=local, delay=11, delays=11/0/0/0.32, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post gruppensprecher) Oct 22 10:40:32 wakis02 postfix/local[4909]: 8010E3A: to=vorstand@localhost, relay=local, delay=11, delays=11/0.01/0/0.32, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post vorstand) Oct 22 10:40:32 wakis02 postfix/local[4902]: 8010E3A: to=ak-leiter@localhost, relay=local, delay=11, delays=11/0/0/0.33, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post ak-leiter) Oct 22 10:40:32 wakis02 postfix/local[4907]: 8010E3A: to=kollegium@localhost, relay=local, delay=11, delays=11/0.01/0/0.33, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post kollegium) Oct 22 10:40:32 wakis02 postfix/local[4905]: 8010E3A: to=beisitz@localhost, relay=local, delay=11, delays=11/0/0/0.34, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post beisitz) Oct 22 10:40:32 wakis02 postfix/smtpd[4898]: 8251C2697: client=localhost[::1] Oct 22 10:40:32 wakis02 postfix/cleanup[4901]: 8251C2697: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:32 wakis02 postfix/qmgr[1304]: 8010E3A: removed Oct 22 10:40:33 wakis02 postfix/smtpd[4920]: disconnect from localhost[::1] Oct 22 10:40:33 wakis02 postfix/smtpd[4920]: connect from localhost[::1] Oct 22 10:40:34 wakis02 postfix/qmgr[1304]: 63AC226AE: from=beisitz-bounces@waldorfkindergarten-erlangen.de, size=314101, nrcpt=2 (queue active) Oct 22 10:40:34 wakis02 postfix/smtpd[4920]: 230E52694: client=localhost[::1] Oct 22 10:40:34 wakis02 postfix/cleanup[4921]: 230E52694: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:38 wakis02 postfix/smtp[4937]: 63AC226AE: to=email1@mail.de, relay=smtp.1und1.de[212.227.15.167]:25, delay=15, delays=11/0.13/0.18/4.2, dsn=2.0.0, status=sent (250 Requested mail action okay, completed: id=0LkPdb-1aMsY51RPP-00cMXf) Oct 22 10:40:38 wakis02 postfix/smtp[4937]: 63AC226AE: to=email2@mail.de, relay=smtp.1und1.de[212.227.15.167]:25, delay=15, delays=11/0.13/0.18/4.2, dsn=2.0.0, status=sent (250 Requested mail action okay, completed: id=0LkPdb-1aMsY51RPP-00cMXf) Oct 22 10:40:38 wakis02 postfix/qmgr[1304]: 63AC226AE: removed Oct 22 10:40:43 wakis02 postfix/qmgr[1304]: 8251C2697: from=alexandrakick@web.de, size=312695, nrcpt=5 (queue active) Oct 22 10:40:43 wakis02 postfix/smtpd[4898]: 98DC6269A: client=localhost[::1] Oct 22 10:40:43 wakis02 postfix/local[4905]: 8251C2697: to=beisitz@localhost, relay=local, delay=12, delays=11/0/0/0.35, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post beisitz) Oct 22 10:40:43 wakis02 postfix/local[4902]: 8251C2697: to=ak-leiter@localhost, relay=local, delay=12, delays=11/0/0/0.35, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post ak-leiter) Oct 22 10:40:43 wakis02 postfix/local[4903]: 8251C2697: to=gruppensprecher@localhost, relay=local, delay=12, delays=11/0/0/0.35, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post gruppensprecher) Oct 22 10:40:43 wakis02 postfix/local[4909]: 8251C2697: to=kollegium@localhost, relay=local, delay=12, delays=11/0/0/0.36, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post kollegium) Oct 22 10:40:43 wakis02 postfix/local[4907]: 8251C2697: to=vorstand@localhost, relay=local, delay=12, delays=11/0/0/0.36, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post vorstand) Oct 22 10:40:43 wakis02 postfix/cleanup[4901]: 98DC6269A: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:44 wakis02 postfix/qmgr[1304]: 8251C2697: removed Oct 22 10:40:45 wakis02 postfix/qmgr[1304]: 230E52694: from=ak-leiter-bounces@waldorfkindergarten-erlangen.de, size=314166, nrcpt=1 (queue active) Oct 22 10:40:46 wakis02 postfix/smtpd[4920]: E1EE526B3: client=localhost[::1] Oct 22 10:40:46 wakis02 postfix/cleanup[4921]: E1EE526B3: message-id=008b01d10ca4$eb5fd0b0$c21f7210$@de Oct 22 10:40:50 wakis02 postfix/smtp[4937]: 230E52694: to=email3@mail.de, relay=smtp.1und1.de[212.227.15.183]:25, delay=17, delays=12/0.32/0.2/4.1, dsn=2.0.0, status=sent (250 Requested mail action okay, completed: id=0MUjgO-1a1LbG1CxA-00Y9t2) Oct 22 10:40:50 wakis02 postfix/qmgr[1304]: 230E52694: removed Oct 22 10:40:54 wakis02 postfix/smtpd[4898]: disconnect from localhost[::1] Oct 22 10:40:54 wakis02 postfix/qmgr[1304]: 98DC6269A: from=alexandrakick@web.de, size=312696, nrcpt=5 (queue active) Oct 22 10:40:55 wakis02 postfix/local[4905]: 98DC6269A: to=gruppensprecher@localhost, relay=local, delay=12, delays=11/0/0/0.38, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post gruppensprecher) Oct 22 10:40:55 wakis02 postfix/local[4907]: 98DC6269A: to=kollegium@localhost, relay=local, delay=12, delays=11/0/0/0.39, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post kollegium) Oct 22 10:40:55 wakis02 postfix/local[4903]: 98DC6269A: to=vorstand@localhost, relay=local, delay=12, delays=11/0.01/0/0.39, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post vorstand) Oct 22 10:40:55 wakis02 postfix/local[4909]: 98DC6269A: to=beisitz@localhost, relay=local, delay=12, delays=11/0/0/0.4, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post beisitz) Oct 22 10:40:55 wakis02 postfix/local[4902]: 98DC6269A: to=ak-leiter@localhost, relay=local, delay=12, delays=11/0/0/0.4, dsn=2.0.0, status=sent (delivered to command: /var/lib/mailman/mail/mailman post ak-leiter) Oct 22 10:40:55 wakis02 postfix/qmgr[1304]: 98DC6269A: removed
On 10/27/2015 12:30 AM, Marco Stoecker wrote:
On 10/26/2015 11:32 PM, Mark Sapiro wrote:
On 10/26/2015 02:14 PM, Marco Stoecker wrote:
I did attach the header from a recent message I got last week. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
But the email, which I sent last week, with the attachement never made it to this list. How can I send the headers instead, cause this email would be very long?
I never saw such a message, even in the moderation queue because of size.
I and the list have now received it. It was sent during the recent server outage on only reached the server earlier today.
It is helpful. I will copy some of the header info below.
...
The most helpful information would be the Postfix log entries from several minutes before Oct 22 10:40:33 2015 up to Oct 22 10:44:28 2015,
This is the related mail.log entry:
Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: connect from localhost[::1] Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: BF9AD1C94:
...
This doesn't start early enough. The duplication occurs at 10:37:47. The duplication occurs because of the way your mail is ultimately delivered to mailman. Here is an excerpt from header1.txt in your other mail:
Received: from imap.1und1.de [212.227.15.188]
by wakis02.local with IMAP (fetchmail-6.3.26)
Thu, 22 Oct 2015 10:40:43 +0200 (CEST)
Received: from [212.227.15.41] ([212.227.15.41]) by mx.kundenserver.de
(mxeue002) with ESMTPS (Nemesis) id 0M1foo-1aeGtV2fLW-00tjIT for
mailmanserver@waldorfkindergarten-erlangen.de; Thu, 22 Oct 2015 10:37:47
+0200
Received: from mout.web.de ([212.227.15.3]) by mx.kundenserver.de (mxeue002)
with ESMTPS (Nemesis) id 0MVE1V-1a1cd42azt-00YP1m for
ak-leiter@waldorfkindergarten-erlangen.de; Thu, 22 Oct 2015 10:37:47 +0200
Received: from Klamotten ([84.168.195.183]) by smtp.web.de (mrweb003) with
ESMTPSA (Nemesis) id 0MTh7A-1Zy14E1g36-00QRsw; Thu, 22 Oct 2015 10:37:47
+0200
From: "Alexandra Kick" alexandrakick@web.de
To: "'Alexandra Kick'" alexandrakick@web.de,
Looking at the Received headers in chronological order (bottom to top), the message is received by smtp.web.de and relayed (as mout.web.de) to mx.kundenserver.de for the ak-leiter@waldorfkindergarten-erlangen.de list. It is then relayed as is probably all list mail on that server to itself for mailmanserver@waldorfkindergarten-erlangen.de. It is then picked up by fetchmail and processed further.
The other messages headers all look the same except they are each initially for one of the other lists, but they all get forwarded to mailmanserver@waldorfkindergarten-erlangen.de.
Now, I'm sure what happens is one message for ak-leiter@waldorfkindergarten-erlangen.de gets to mailmanserver@waldorfkindergarten-erlangen.de and at some point later the fact that it was originally received just for the ak-leiter@waldorfkindergarten-erlangen.de list is lost or ignored and the process looks at the To: header of the mail, sees 5 lists and forwards the mail to all 5 lists.
The same thing happens with each of the other 4 messages resulting in 5 messages to each of the 5 lists.
The answer is your process for delivering mail to Mailman is flawed because it takes a message received for one list only and delivers it to every list mentioned in To: (and maybe Cc:).
On 10/28/2015 05:49 AM, Mark Sapiro wrote:
On 10/27/2015 12:30 AM, Marco Stoecker wrote:
On 10/26/2015 11:32 PM, Mark Sapiro wrote:
On 10/26/2015 02:14 PM, Marco Stoecker wrote:
I did attach the header from a recent message I got last week. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
But the email, which I sent last week, with the attachement never made it to this list. How can I send the headers instead, cause this email would be very long?
I never saw such a message, even in the moderation queue because of size.
I and the list have now received it. It was sent during the recent server outage on only reached the server earlier today.
It is helpful. I will copy some of the header info below.
...
The most helpful information would be the Postfix log entries from several minutes before Oct 22 10:40:33 2015 up to Oct 22 10:44:28 2015,
This is the related mail.log entry:
Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: connect from localhost[::1] Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: BF9AD1C94:
...
This doesn't start early enough. The duplication occurs at 10:37:47. The duplication occurs because of the way your mail is ultimately delivered to mailman. Here is an excerpt from header1.txt in your other mail:
The entry before this in /var/log/mail.log has a time stamp of 09:35:59 which is far before the duplicated message was sent.
Received: from imap.1und1.de [212.227.15.188]
by wakis02.local with IMAP (fetchmail-6.3.26)
Thu, 22 Oct 2015 10:40:43 +0200 (CEST)
Received: from [212.227.15.41] ([212.227.15.41]) by mx.kundenserver.de
(mxeue002) with ESMTPS (Nemesis) id 0M1foo-1aeGtV2fLW-00tjIT for
mailmanserver@waldorfkindergarten-erlangen.de; Thu, 22 Oct 2015 10:37:47
+0200
Received: from mout.web.de ([212.227.15.3]) by mx.kundenserver.de (mxeue002)
with ESMTPS (Nemesis) id 0MVE1V-1a1cd42azt-00YP1m for
ak-leiter@waldorfkindergarten-erlangen.de; Thu, 22 Oct 2015 10:37:47 +0200
Received: from Klamotten ([84.168.195.183]) by smtp.web.de (mrweb003) with
ESMTPSA (Nemesis) id 0MTh7A-1Zy14E1g36-00QRsw; Thu, 22 Oct 2015 10:37:47
+0200
From: "Alexandra Kick" alexandrakick@web.de
To: "'Alexandra Kick'" alexandrakick@web.de,
Looking at the Received headers in chronological order (bottom to top), the message is received by smtp.web.de and relayed (as mout.web.de) to mx.kundenserver.de for the ak-leiter@waldorfkindergarten-erlangen.de list. It is then relayed as is probably all list mail on that server to itself for mailmanserver@waldorfkindergarten-erlangen.de. It is then picked up by fetchmail and processed further.
The other messages headers all look the same except they are each initially for one of the other lists, but they all get forwarded to mailmanserver@waldorfkindergarten-erlangen.de.
Now, I'm sure what happens is one message for ak-leiter@waldorfkindergarten-erlangen.de gets to mailmanserver@waldorfkindergarten-erlangen.de and at some point later the fact that it was originally received just for the ak-leiter@waldorfkindergarten-erlangen.de list is lost or ignored and the process looks at the To: header of the mail, sees 5 lists and forwards the mail to all 5 lists.
The same thing happens with each of the other 4 messages resulting in 5 messages to each of the 5 lists.
The answer is your process for delivering mail to Mailman is flawed because it takes a message received for one list only and delivers it to every list mentioned in To: (and maybe Cc:).
I also posted this problem to the postfix mailing list and today I got an indicator that maybe fetchmail is the root of the problem due to some probably missing multi-drop support, which I have to double check now. But the fetchmail version which is delivered since debian 7 does not support time stamp information in the log.
Thx for your support! Marco
On 10/28/2015 08:57 AM, Marco Stoecker wrote:
On 10/28/2015 05:49 AM, Mark Sapiro wrote:
On 10/27/2015 12:30 AM, Marco Stoecker wrote:
On 10/26/2015 11:32 PM, Mark Sapiro wrote:
On 10/26/2015 02:14 PM, Marco Stoecker wrote:
I did attach the header from a recent message I got last week. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
But the email, which I sent last week, with the attachement never made it to this list. How can I send the headers instead, cause this email would be very long?
I never saw such a message, even in the moderation queue because of size.
I and the list have now received it. It was sent during the recent server outage on only reached the server earlier today.
It is helpful. I will copy some of the header info below.
...
The most helpful information would be the Postfix log entries from several minutes before Oct 22 10:40:33 2015 up to Oct 22 10:44:28 2015,
This is the related mail.log entry:
Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: connect from localhost[::1] Oct 22 10:40:05 wakis02 postfix/smtpd[4898]: BF9AD1C94:
...
This doesn't start early enough. The duplication occurs at 10:37:47. The duplication occurs because of the way your mail is ultimately delivered to mailman. Here is an excerpt from header1.txt in your other mail:
The entry before this in /var/log/mail.log has a time stamp of 09:35:59 which is far before the duplicated message was sent.
Received: from imap.1und1.de [212.227.15.188]
by wakis02.local with IMAP (fetchmail-6.3.26)
Thu, 22 Oct 2015 10:40:43 +0200 (CEST)
Received: from [212.227.15.41] ([212.227.15.41]) by mx.kundenserver.de
(mxeue002) with ESMTPS (Nemesis) id 0M1foo-1aeGtV2fLW-00tjIT for
mailmanserver@waldorfkindergarten-erlangen.de; Thu, 22 Oct 2015 10:37:47
+0200
Received: from mout.web.de ([212.227.15.3]) by mx.kundenserver.de (mxeue002)
with ESMTPS (Nemesis) id 0MVE1V-1a1cd42azt-00YP1m for
ak-leiter@waldorfkindergarten-erlangen.de; Thu, 22 Oct 2015 10:37:47 +0200
Received: from Klamotten ([84.168.195.183]) by smtp.web.de (mrweb003) with
ESMTPSA (Nemesis) id 0MTh7A-1Zy14E1g36-00QRsw; Thu, 22 Oct 2015 10:37:47
+0200
From: "Alexandra Kick" alexandrakick@web.de
To: "'Alexandra Kick'" alexandrakick@web.de,
Looking at the Received headers in chronological order (bottom to top), the message is received by smtp.web.de and relayed (as mout.web.de) to mx.kundenserver.de for the ak-leiter@waldorfkindergarten-erlangen.de list. It is then relayed as is probably all list mail on that server to itself for mailmanserver@waldorfkindergarten-erlangen.de. It is then picked up by fetchmail and processed further.
The other messages headers all look the same except they are each initially for one of the other lists, but they all get forwarded to mailmanserver@waldorfkindergarten-erlangen.de.
Now, I'm sure what happens is one message for ak-leiter@waldorfkindergarten-erlangen.de gets to mailmanserver@waldorfkindergarten-erlangen.de and at some point later the fact that it was originally received just for the ak-leiter@waldorfkindergarten-erlangen.de list is lost or ignored and the process looks at the To: header of the mail, sees 5 lists and forwards the mail to all 5 lists.
The same thing happens with each of the other 4 messages resulting in 5 messages to each of the 5 lists.
The answer is your process for delivering mail to Mailman is flawed because it takes a message received for one list only and delivers it to every list mentioned in To: (and maybe Cc:).
I also posted this problem to the postfix mailing list and today I got an indicator that maybe fetchmail is the root of the problem due to some probably missing multi-drop support, which I have to double check now. But the fetchmail version which is delivered since debian 7 does not support time stamp information in the log.
Thx for your support! Marco
Dear Mark,
I think I do now understand why duplication is happening. But have no idea how to avoid it. So here is how I think it happened.
I send an email to list1 and list2 now on my ISP there is an email forwarding from list1@... and list2@... to the email account mailmansrv@mydomain. So here I have 2 emails in that account which are both to: list1 and list2. Fetchmail now will get these 2 messages with to: to both lists and even postfix gets 2 messages with each adressed to two lists. I think this is the reason why duplication happens. But now I have no glue how to configure on the ISP side or postfix(relayhost) or mailman to avoid such duplicates. Any assistance possible here in this mailing list?
BR Marco
In a message of Wed, 28 Oct 2015 10:22:50 +0100, Marco Stoecker writes:
But now I have no glue how to configure on the ISP side or postfix(relayhost) or mailman to avoid such duplicates. Any assistance possible here in this mailing list?
BR Marco
There is a fetchmail-users mailing list here: https://lists.sourceforge.net/lists/listinfo/fetchmail-users if you need to ask about how to configure fetchmail. I am not sure that is your problem, though.
Laura
On 10/28/2015 02:22 AM, Marco Stoecker wrote:
I think I do now understand why duplication is happening. But have no idea how to avoid it. So here is how I think it happened.
I send an email to list1 and list2 now on my ISP there is an email forwarding from list1@... and list2@... to the email account mailmansrv@mydomain. So here I have 2 emails in that account which are both to: list1 and list2. Fetchmail now will get these 2 messages with to: to both lists and even postfix gets 2 messages with each adressed to two lists. I think this is the reason why duplication happens. But now I have no glue how to configure on the ISP side or postfix(relayhost) or mailman to avoid such duplicates. Any assistance possible here in this mailing list?
Yes. That is what's happening
To fix this, if it is possible to eliminate fetchmail and have the ISP email forwarding go directly to your machine, you can then configure Postfix on your machine to deliver to Mailman with aliases in the normal way and the problem is solved.
If it is not possible to to eliminate fetchmail because, e.g., it is not possible to mail directly to your machine from the ISP, then perhaps you can eliminate the forwarding from list1@... and list2@... to the email account mailmansrv@mydomain and have fetchmail pick up the mail individually per list and deliver only to that list.
Your basic problem is that mailman is designed to run on the server that receives the list mail directly. Because of whatever restrictions exist that prevent that, you (or someone) have come up with this fetchmail kludge and it doesn't work because of the duplication.
Others have made this work by having separate mailboxes on the host for each list, list-owner, list-subscribe, etc. address and using fetchmail to deliver mailbox by mailbox to the appropriate place on the Mailman machine.
On 10/28/2015 04:22 PM, Mark Sapiro wrote:
On 10/28/2015 02:22 AM, Marco Stoecker wrote:
I think I do now understand why duplication is happening. But have no idea how to avoid it. So here is how I think it happened.
I send an email to list1 and list2 now on my ISP there is an email forwarding from list1@... and list2@... to the email account mailmansrv@mydomain. So here I have 2 emails in that account which are both to: list1 and list2. Fetchmail now will get these 2 messages with to: to both lists and even postfix gets 2 messages with each adressed to two lists. I think this is the reason why duplication happens. But now I have no glue how to configure on the ISP side or postfix(relayhost) or mailman to avoid such duplicates. Any assistance possible here in this mailing list?
Yes. That is what's happening
To fix this, if it is possible to eliminate fetchmail and have the ISP email forwarding go directly to your machine, you can then configure Postfix on your machine to deliver to Mailman with aliases in the normal way and the problem is solved.
my machine has dynamic/changing ip adresses, as it is connected to the internet via dial-in connection.
If it is not possible to to eliminate fetchmail because, e.g., it is not possible to mail directly to your machine from the ISP, then perhaps you can eliminate the forwarding from list1@... and list2@... to the email account mailmansrv@mydomain and have fetchmail pick up the mail individually per list and deliver only to that list.
I have 24 lists and only 20 email accounts to configure separately.
Your basic problem is that mailman is designed to run on the server that receives the list mail directly. Because of whatever restrictions exist that prevent that, you (or someone) have come up with this fetchmail kludge and it doesn't work because of the duplication.
I'm also not happy with this, but the kindergarden for which the solution is, can afford only this.
Others have made this work by having separate mailboxes on the host for each list, list-owner, list-subscribe, etc. address and using fetchmail to deliver mailbox by mailbox to the appropriate place on the Mailman machine.
problem solved :-D I got the indication of envelope 1 Received to skip the first Received line + 'aka mx.kundenserver.de' in the fetchmailrc was the solution to get the envelope information of to what list the email was intended to. So I can keep fetchmail, cause there is no other solution so far.
Thx for your support! Marco
On 10/28/2015 05:45 PM, Marco Stoecker wrote:
problem solved :-D I got the indication of envelope 1 Received to skip the first Received line + 'aka mx.kundenserver.de' in the fetchmailrc was the solution to get the envelope information of to what list the email was intended to. So I can keep fetchmail, cause there is no other solution so far.
Yes. There's no inherent problem with using fetchmail per se in an environment like yours as long as fetchmail can be told to which address any particular message was originally sent and delivers it to only that list and not blindly to every list mentioned in To:
On 10/28/2015 12:57 AM, Marco Stoecker wrote:
On 10/28/2015 05:49 AM, Mark Sapiro wrote:
This doesn't start early enough. The duplication occurs at 10:37:47. The duplication occurs because of the way your mail is ultimately delivered to mailman. Here is an excerpt from header1.txt in your other mail:
The entry before this in /var/log/mail.log has a time stamp of 09:35:59 which is far before the duplicated message was sent.
Yes, they are on another machine.
On 10/21/2015 11:45 PM, Mark Sapiro wrote:
On 10/21/2015 02:14 PM, Marco Stoecker wrote:
How do I identify whether the message comes from mailman or list1 or list2? Sorry for the questions, I'm not that experienced so far.
Here are some of the headers of your message as I received it from the mailman-users list with my comments interspersed. How you actually see these headers depends on your mail client. It may be something like "show original" or "view source" or something else depending on the client. I have removed some headers which aren't relevant to this discussion.
<snipped>
If you just get the complete headers from all the duplicates of one message received by one user and post those, I can help interpret them.
This would be very great! I attach the header from a recent message I got today. I'm a member of a list and the sender sent this mail to 5 mailinglists on our server. Each member of these 5 lists got 5 messages.
Did I understood right, all entry in /var/log/mailman/smtp file are made by mailman and in this case it could be an indicator that mailman send the email 5 times to each list?
And here is the corresponding entry in the /var/log/mailman/smtp file:
Oct 22 10:40:33 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 21.833 seconds Oct 22 10:41:02 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 28.426 seconds Oct 22 10:41:12 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.447 seconds Oct 22 10:41:23 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 11.236 seconds Oct 22 10:41:29 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 5.095 seconds Oct 22 10:41:39 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.615 seconds Oct 22 10:41:50 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.871 seconds Oct 22 10:41:55 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 4.794 seconds Oct 22 10:42:06 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.890 seconds Oct 22 10:42:22 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 15.688 seconds Oct 22 10:42:32 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 10.289 seconds Oct 22 10:42:43 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.736 seconds Oct 22 10:42:53 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.497 seconds Oct 22 10:43:09 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 16.035 seconds Oct 22 10:43:15 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 5.468 seconds Oct 22 10:43:25 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 10.680 seconds Oct 22 10:43:31 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 5.253 seconds Oct 22 10:43:46 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 15.867 seconds Oct 22 10:43:57 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.956 seconds Oct 22 10:44:09 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 11.101 seconds Oct 22 10:44:18 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to gruppensprecher for 6 recips, completed in 9.922 seconds Oct 22 10:44:28 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to beisitz for 3 recips, completed in 9.971 seconds Oct 22 10:44:39 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to kollegium for 16 recips, completed in 10.315 seconds Oct 22 10:44:55 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to ak-leiter for 13 recips, completed in 16.343 seconds Oct 22 10:45:00 2015 (897) 008b01d10ca4$eb5fd0b0$c21f7210$@de smtp to vorstand for 3 recips, completed in 4.947 seconds
On 10/21/2015 02:40 AM, Mark Sapiro wrote:
On 10/20/2015 04:45 PM, Marco Stoecker wrote:
I see a pattern now, when duplicates occur and it may have to do with mailman. If someone in a list send an email to two or three lists on the same server, the recipients of the list will get two or three times the same email. If it is sent to one list only, no duplicate occur. Any glue what the root cause could be?
If you are saying that people who are members of more than one list receive a copy from each list of which they are a member when a post is sent to multiple lists, that's the way Mailman works.
I.e. If I am a member of list1, list2 and list3 and someone posts to all three lists, I will receive a copy from each list.
The list member setting "Avoid duplicate copies of messages?" (nodupes) does not affect this. It only affects whether I receive a list copy if I am also a To: or Cc: addressee of the post.
The list's Non-digest options -> regular_exclude_lists can modify this behavior somewhat. E.g., if the regular_exclude_lists setting for list1@example.com includes list2@example.com and list3@example.com and the regular_exclude_lists setting for list2@example.com includes list3@example.com, then a member of all three lists will receive a copy of a post sent to all three lists from only list3@example.com.
regular_exclude_lists need to be set up with care. In particular, if list2@example.com is in list1@example.com's regular_exclude_lists, list1@example.com MUST not be in list2@example.com's regular_exclude_lists or a post sent to both lists will not be received by anyone who is a member of both lists.
Note that in the above, member means non-digest member. Digest members are not affected by this.
Dear Mark,
in addition to my former post, where do I find information about how mailman is handing over to postfix. Mayby that could help me.
BR Marco
participants (3)
-
Laura Creighton -
Marco Stoecker -
Mark Sapiro