Part of moderated mails postponed to later digests
One of my list administrators complains that digest mails now seems to be sent in fragmented chunks and instead of sending all moderated mails in a digest the same day, only some of the mails will be sent with a digest the same day, and the rest will be sent in a digest with the next moderated mails, half a week later. For instance, when moderating say 50 mails in week 1 and 50 mails in week 2, the digest mail in week 1 will contain say 30 mails in a digest, while the digest in week 2 will contain like 20 mails from week 1 and like 10 mails from week 2.
There is no information about when the admin discovered the problem, but on August 20th 2015 Mailman has been upgraded from 3:2.1.12-18.el6 to 3:2.1.12-25.el6 and I wonder if the DMARC implementation in 3:2.1.12-22.el6 has something to do with it, as this is the only changes I can think of having any effect on the mail behavior.
dmarc_quarantine_moderation_action is set to Yes on the list in particular and according to some discussions on the Internet, this strict action has been coursing bounces, so I suggested him to set this to No.
But even being strict, I don't see why this would course parts of the digest being held half a week, so something else must interfere.
Regards Henrik Rasmussen
On 11/03/2015 06:32 AM, Henrik Rasmussen wrote:
One of my list administrators complains that digest mails now seems to be sent in fragmented chunks and instead of sending all moderated mails in a digest the same day, only some of the mails will be sent with a digest the same day, and the rest will be sent in a digest with the next moderated mails, half a week later. For instance, when moderating say 50 mails in week 1 and 50 mails in week 2, the digest mail in week 1 will contain say 30 mails in a digest, while the digest in week 2 will contain like 20 mails from week 1 and like 10 mails from week 2.
My best guess is you are triggering digests on size (Digest options -> digest_size_threshold) and either not periodically (digest_send_periodic = No) or cron/senddigests runs only weekly.
50 messages are approved; the 30th message triggers a digest on size. And the remaining wait until digest_size_threshold is reached again. Then the next week a bunch more messages are approved and the 10th message triggers a digest on size and the rest wait and until either threshold is reached again or a periodic digest is sent.
As to why this is changed behavior, perhaps after the upgrade, cron/senddigests is not being run for some reason to produce a daily digest. Check Mailman's crontab, Mailman's error log and the system cron log.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
My best guess is you are triggering digests on size (Digest options -> digest_size_threshold) and either not periodically (digest_send_periodic = No) or cron/senddigests runs only weekly.
To clarify, he has set digest_size_threshold to 10000 and digest_send_periodic =Yes. /usr/lib/mailman/cron/senddigests is run by cron every day at noon.
Other digest settings is
nondigestable = True scrub_nondigest = False digestable = True digest_is_default = 1 mime_is_default_digest = 0 digest_size_threshhold = 10000 digest_send_periodic = 1 digest_volume_frequency = 3 discard_these_nonmembers = (241 members) ban_list = (188 members)
The digest_size_threshhold = 10000 may likely course the problem, but is necessary as a digest containing a single mail is sent each time a mail is sent to the list-address as suggested by https://mail.python.org/pipermail/mailman-users/2014-August/077622.html.
Regards Henrik
On 11/04/2015 02:09 AM, Henrik Rasmussen wrote:
My best guess is you are triggering digests on size (Digest options -> digest_size_threshold) and either not periodically (digest_send_periodic = No) or cron/senddigests runs only weekly.
To clarify, he has set digest_size_threshold to 10000 and digest_send_periodic =Yes. /usr/lib/mailman/cron/senddigests is run by cron every day at noon.
Other digest settings is
nondigestable = True scrub_nondigest = False digestable = True digest_is_default = 1 mime_is_default_digest = 0 digest_size_threshhold = 10000 digest_send_periodic = 1 digest_volume_frequency = 3 discard_these_nonmembers = (241 members) ban_list = (188 members)
The digest_size_threshhold = 10000 may likely course the problem, but is necessary as a digest containing a single mail is sent each time a mail is sent to the list-address as suggested by https://mail.python.org/pipermail/mailman-users/2014-August/077622.html.
That's a bug <https://bugs.launchpad.net/mailman/+bug/558274> which was fixed in MM 2.1.16, but in any case, if you don't want digests triggered on size and your mailman is older than 2.1.16, setting digest_size_threshhold = 10000 (KB = 10 MB) should stop digests being triggered on size unless your list has a lot of large graphics attached to posts. I.e. 10MB / 100 posts is still 100KB per post.
Anyway, I understood your OP to say digests were being sent only weekly when batches of held messages are approved. If as you say, digest_send_periodic =Yes and /usr/lib/mailman/cron/senddigests is run by cron every day at noon, even if 50 messages are approved and 30 of them are sent immediately in a digest (triggered on size), the other 20 should be sent in a digest the next noon.
If this is not the case, either cron/senddigests is not running every noon or it is somehow skipping this list or encountering errors with this list. Is there anything in Mailman's error log?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Sorry for the late reply.
Mark Sapiro [mailto:mark@msapiro.net] Sent: 4. november 2015 17:07
Anyway, I understood your OP to say digests were being sent only weekly when batches of held messages are approved. That was an example of the frequency, held mails is being approved by the moderator. When the moderator approves a bunch of mails in one week, the list admin claims that only some of the mails are posted to the list, but that some of the mails will not be sent in a digest until the next moderation a week later (example).
if you don't want digests triggered on size The list admin doesn't need digests sent according to size so a digest_size_threshhold = 10000 should be fine. The moderator wants to have the mails included in a digest the next noon after approving the mails.
and your mailman is older than 2.1.16 I run Mailman 2.1.12 which is the latest available for yum update on RHEL6. I think the last Mailman version on RHEL7 is 2.1.15, but we're not there yet.
I just confirmed that senddigest is run every noon: /var/log/cron:Nov 8 12:00:01 p1kitlst01l CROND[18196]: (mailman) CMD (/usr/lib/mailman/cron/senddigests) /var/log/cron:Nov 9 12:00:01 p1kitlst01l CROND[27052]: (mailman) CMD (/usr/lib/mailman/cron/senddigests)
If this is not the case, either cron/senddigests is not running every noon or it is somehow skipping this list or encountering errors with this list. Is there anything in Mailman's error log?
The error log doesn't show anything in association with the list in question.
According to the vette log, some are discarded by moderator actions, : Oct 25 10:20:42 2015 (16801) Precedence: bulk message discarded by: LISTNAME-request@list.example.com And of cause a lot of "Reason: Non-members" and spam-like messages "Reason: Your message was deemed inappropriate by the moderator.".
Regards, Henrik Rasmussen
-----Oprindelig meddelelse----- Fra: Mark Sapiro [mailto:mark@msapiro.net] Sendt: 4. november 2015 17:07 Til: Henrik Rasmussen; mailman-users@python.org Emne: Re: SV: [Mailman-Users] Part of moderated mails postponed to later digests
On 11/04/2015 02:09 AM, Henrik Rasmussen wrote:
My best guess is you are triggering digests on size (Digest options -> digest_size_threshold) and either not periodically (digest_send_periodic = No) or cron/senddigests runs only weekly.
To clarify, he has set digest_size_threshold to 10000 and digest_send_periodic =Yes. /usr/lib/mailman/cron/senddigests is run by cron every day at noon.
Other digest settings is
nondigestable = True scrub_nondigest = False digestable = True digest_is_default = 1 mime_is_default_digest = 0 digest_size_threshhold = 10000 digest_send_periodic = 1 digest_volume_frequency = 3 discard_these_nonmembers = (241 members) ban_list = (188 members)
The digest_size_threshhold = 10000 may likely course the problem, but is necessary as a digest containing a single mail is sent each time a mail is sent to the list-address as suggested by https://mail.python.org/pipermail/mailman-users/2014-August/077622.html.
That's a bug <https://bugs.launchpad.net/mailman/+bug/558274> which was fixed in MM 2.1.16, but in any case, if you don't want digests triggered on size and your mailman is older than 2.1.16, setting digest_size_threshhold = 10000 (KB = 10 MB) should stop digests being triggered on size unless your list has a lot of large graphics attached to posts. I.e. 10MB / 100 posts is still 100KB per post.
Anyway, I understood your OP to say digests were being sent only weekly when batches of held messages are approved. If as you say, digest_send_periodic =Yes and /usr/lib/mailman/cron/senddigests is run by cron every day at noon, even if 50 messages are approved and 30 of them are sent immediately in a digest (triggered on size), the other 20 should be sent in a digest the next noon.
If this is not the case, either cron/senddigests is not running every noon or it is somehow skipping this list or encountering errors with this list. Is there anything in Mailman's error log?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11/9/15 4:50 AM, Henrik Rasmussen wrote:
Mark Sapiro [mailto:mark@msapiro.net] Sent: 4. november 2015 17:07
Anyway, I understood your OP to say digests were being sent only weekly when batches of held messages are approved. That was an example of the frequency, held mails is being approved by the moderator. When the moderator approves a bunch of mails in one week, the list admin claims that only some of the mails are posted to the list, but that some of the mails will not be sent in a digest until the next moderation a week later (example).
This is the first time (I think) you mentioned 'posted to the list' Are the messages missing from the digest not posted to the list at all until the next week? I.e. are they not in the list archives and not delivered to any non-digest members?
If in fact the missing messages don't appear in the archive until the next week, this is a different issue which has nothing to do with digests per se.
if you don't want digests triggered on size The list admin doesn't need digests sent according to size so a digest_size_threshhold = 10000 should be fine. The moderator wants to have the mails included in a digest the next noon after approving the mails.
Are the messages which are sent, sent in an immediate digest at the time of approval or just the next noon?
If the former, that digest is triggered on size. If the latter, those are the only messages that actually got approved at that time.
and your mailman is older than 2.1.16 I run Mailman 2.1.12 which is the latest available for yum update on RHEL6. I think the last Mailman version on RHEL7 is 2.1.15, but we're not there yet.
So you can't use digest_size_threshhold = 0 for unlimited because of the bug, but if digests are triggered on size and you don't want that, you can set digest_size_threshhold to a bigger number.
According to the vette log, some are discarded by moderator actions, : Oct 25 10:20:42 2015 (16801) Precedence: bulk message discarded by: LISTNAME-request@list.example.com And of cause a lot of "Reason: Non-members" and spam-like messages "Reason: Your message was deemed inappropriate by the moderator.".
What does the vette log say about the messages that don't appear in a digest until the following week? Do they show as approved at the same time as the others that are in the first digest or not until later.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
This is the first time (I think) you mentioned 'posted to the list' This may be something with lost in translation. What I meant by "posted to the list" is "forwarded to the members" after approval.
Are the messages missing from the digest not posted to the list at all until the next week? I.e. are they not in the list archives and not delivered to any non-digest members? There are only a few non-digest members and the list admin is not aware whether they receive the mails or not.
If in fact the missing messages don't appear in the archive until the next week, this is a different issue which has nothing to do with digests per se. I will look closely at that, and get info from the list admin on when he approves mails.
if you don't want digests triggered on size The list admin doesn't need digests sent according to size so a digest_size_threshhold = 10000 should be fine. The moderator wants to have the mails included in a digest the next noon after approving the mails.
Are the messages which are sent, sent in an immediate digest at the time of approval or just the next noon? If the former, that digest is triggered on size. If the latter, those are the only messages that actually got approved at that time. Next noon.
What does the vette log say about the messages that don't appear in a digest until the following week? Do they show as approved at the same time as the others that are in the first digest or not until later. I am still trying to figure that out, as I still need to know exactly when the list admin has approved messages. The vette log shows a lot of Refused posting labeled with the list name, but all the held message approved messages does not contain information on which list the message was approved, so I can't use the vette log to determine when messages on the list was approved.
But I wonder why the smtp log contains so many entries postings to exactly 1209 recipients on the list, even the list has 1222 normal members and 6493 digest members. Why only 6 times 1209. And are the 1209 digest members while the 6 recips could be regular members or something?
/var/log/mailman/smtp Nov 09 09 06 44 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 1 recips, completed in 0.098 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/post Nov 09 13 28 33 2015 (16804) post to LISTNAME from someone@example.com, size=5675, message-id=<6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local>, success
Where can I see which messages was contained in which digests and when? My problem is that I don't seem to be able to tell whether the list admin is right or not. Not that I doubt his statement, but I really like to be able to tell when what happens.
Regards, Henrik Rasmussen
On 11/12/15 5:06 AM, Henrik Rasmussen wrote:
What does the vette log say about the messages that don't appear in a digest until the following week? Do they show as approved at the same time as the others that are in the first digest or not until later. I am still trying to figure that out, as I still need to know exactly when the list admin has approved messages. The vette log shows a lot of Refused posting labeled with the list name, but all the held message approved messages does not contain information on which list the message was approved, so I can't use the vette log to determine when messages on the list was approved.
Yes, that is a bug, <https://bugs.launchpad.net/mailman/+bug/1295875>, fixed in MM 2.1.18.
But I wonder why the smtp log contains so many entries postings to exactly 1209 recipients on the list, even the list has 1222 normal members and 6493 digest members. Why only 6 times 1209. And are the 1209 digest members while the 6 recips could be regular members or something?
/var/log/mailman/smtp Nov 09 09 06 44 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 1 recips, completed in 0.098 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds /var/log/mailman/smtp Nov 09 09 06 45 2015 (16804) <mailman.577.1447056402.16802.LISTNAME@LISTSERVER.dk> smtp to LISTNAME for 6 recips, completed in 0.200 seconds
The above are a Mailman notice. It is unclear to me exactly what is happening here? The first log message is the message to LISTNAME-owner and the next is the resending of that message to the 6 owner/moderator addresses. The next 5 appear to be duplicates because the 0.200 seconds are identical and even if there were 6 separate sends, that totals 1.2 seconds, yet the timestamp never advanced even 1 second.
/var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds /var/log/mailman/smtp Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds
Here again this appears to be 6 duplicate copies of one message. It could be 6 approved posts each delivered to 1209 eligible non-digest members upon approval (the other 13 non-digest members could be delivery disabled or direct addressees with 'no dups' or a combination), but it is not 6 posts because the log messages are all for the same Message-ID and all from the same (pid = 16804) OutgoingRunner and delivery takes over 110 seconds, but the timestamp doesn't advance. Also, 6 separate sends to 1209 recips would be very unlikely to be completed in the exact same time of 110.189 seconds.
/var/log/mailman/post Nov 09 13 28 33 2015 (16804) post to LISTNAME from someone@example.com, size=5675, message-id=<6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local>, success
And the post log shows only one post.
Where can I see which messages was contained in which digests and when? My problem is that I don't seem to be able to tell whether the list admin is right or not. Not that I doubt his statement, but I really like to be able to tell when what happens.
The send of the digests would be logged as above in the smtp log at 'Nov 09 12 xx xx'. To see which messages it contained, you'd have to see the digest itself. That info isn't logged.
You could subscribe yourself to the MIME digest. Then you'd see the messages in it and their Message-IDs. The Message-IDs should be in the vette log. Also, for the smtp log entry above like
Nov 09 13 28 33 2015 (16804) <6f7a7eb8e5e442668ea1a864e84c2530@pobox.engr.local> smtp to LISTNAME for 1209 recips, completed in 110.189 seconds
there should be a vette log entry with the same Message-ID and a timestamp about 110-115 seconds earlier.
The six apparent duplications of certain smtp log entries is very puzzling. I don't know what would cause that.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Henrik Rasmussen -
Mark Sapiro