Delivery errors and spam grading
Hi Friends,
I have a few doubts, I write you in the hope of a hint. For Mailman 2.1.23 on Debian Stretch:
- How is it possible from Mailman monitor the delivery errors? Only sysadmin mail server from the logs?
- Is it possible avoid spam grading, increasing the number of sending sessions/decreasing individual emails sent per session? And eventually your hint? :-)
Many thanks!
Davide
-- cosmogoniA n o p r o v a r e n o f a r e o n o n f a r e n o n c e p r o v a r e
On 3/18/2019 4:51 AM, Davide Marchi wrote:
This is what VERP deliveries are for. The FROM address is tagged so that bounces can be clearly identified.
https://wiki.list.org/DOC/So%20what%20is%20this%20VERP%20stuff
david
-- IBM i on Power Systems: For when you can't afford to be out of business!
I'm riding 615 miles (Yes, you read that right) in the American Diabetes Association's Tour de Cure to raise money for diabetes research, education, advocacy, and awareness. You can make a tax-deductible donation to my ride by visiting https://mideml.diabetessucks.net.
You can see where my donations come from by visiting my interactive donation map ... https://mideml.diabetessucks.net/map (it's a geeky thing).
I may have diabetes, but diabetes doesn't have me!
Davide Marchi writes:
That depends on what you mean by "monitor" and "delivery errors". If you mean mail refused by some system (typically the final recipient) and returned to Mailman, that is what we call "bounces". Mailman can be configured on the [Bounce processing] screen to send mail to the list owner (only, not moderators or the site administrator) four types of notifications about bounces. The default is usually appropriate.
In some cases recipient systems don't identify the bounced recipient. In that case you can use VERP as described below, which arranges that Mailman receives all bounces in such a way that the recipient can be identified (a special-purpose return address is allocated per recipient).
There is no web-based monitor in Mailman 2. These notifications are sent by email to the list owner. However, if your email provider uses a web-based manager like cPanel, you may have some access to this information (I don't use cPanel, so I don't know what features it provides for monitoring system health).
If you mean any other kind of delivery failure that is detected by the Mailman host, but is not returned to Mailman, then it is simply impossible for Mailman to even be aware of them, and you would have to check the MTA (mail server) logs. And of course there are delivery errors that happen and never are returned to the Mailman host at all (you find out about these because subscribers tell you). There's nothing that can be done about those by Mailman, either.
You should set DMARC Moderation Action to Munge From in the [Privacy options -- Sender filters] screen. (Strictly speaking this isn't spam filtering, it's mitigating breakage induced by incompetent email providers.) And of course you should filter incoming mail for spam (and if you have any human users besides root on the Mailman host, outgoing mail too -- this is why it's a good idea to have a dedicated host for Mailman!) The MTA should be set up to use DKIM and SPF.
increasing the number of sending sessions/decreasing individual emails sent per session?
It's not obvious to me that this is very useful: most of the big providers count the number of times a message is received by any method. They don't care if you do it one at a time (although they may shut you down before any mail is delivered if they are counting the recipients per session). The usual reason for throttling is because Mailman's provider doesn't like accepting huge numbers of addressees from Mailman, not because you have tons of subscribers at one provider.
Only very crudely, in a site-wide way or very drastically per-list. The site-wide option works with Mailman's internal "SMTPDirect" DELIVERY_MODULE (which is the default). Set SMTP_MAX_RCPTS to some "reasonable" figure in mm_cfg.py. This applies to all outgoing connections.
Alternatively, you can set VERP_INTERVAL=1 in mm_cfg.py, which ensures that every message is a one-subscriber-per-SMTP-session sitewide, or for each list you can enable Personalization in the [Non-digest options] which has the same effect for one list. (You probably should also disable the Digestable option in [Digest options].
Steve
On 3/18/19 5:39 AM, Stephen J. Turnbull wrote:
However errors that occur during SMTP from Mailman to the outgoing MTA are returned to Mailman. Mailman sees the SMTP status return and if it is a 5xx status, scores that as a bounce and if it is a 4xx status retries sending the message to the failed recipient(s) at configurable intervals for a configurable time.
The retry interval is DELIVERY_RETRY_WAIT with a default of one hour, but in MM 2.1.23, DELIVERY_RETRY_WAIT is ignored and the interval is 15 minutes.
The duration of retries is DELIVERY_RETRY_PERIOD with a default of 5 days, after which Mailman gives up without scoring a bounce.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Mar 18, 2019 at 4:39 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
(You probably should also disable the Digestable option in [Digest options]
I'm also receiving several bounces back for errors such as: "SMTP error from remote mail server after end of data: 554 Failed: Malformed MIME field: X-Ham-Report:"
Or: "SMTP error from remote mail server after end of data: 554 5.7.1 [P4] Message blocked due to spam content in the message."
Mostly associated with the digest. Curious about the statement above from Steve: is disabling the digest option something people do regularly to prevent issues like this? and if so - if I were to disable it now, what happens to those members who are using that option? I assume they just start automatically receiving non-digest messages, but figured I should ask.
thanks, JD
On 3/18/2019 8:44 PM, Jim Dory wrote:
The second "error" message is one over which you have little control. The recipient's mail system detected content in the digest that it considered spam. I assume that if the messages in the digest had been sent individually in non-digest mode, then the message that had the objectionable content would have been similarly flagged.
As for the first "error" - A quick Google search showed that the
X-Ham-Report:header line is written by Cpanel when it detects that the message is not spam. I do not know if any other software writes this line. I am not an expert in MIME, but I do not see how any "X-" header line could be treated as a malformed MIME header.
If you have users who are using digest mode, then removing that option will cause consternation with those users. And I doubt that the digest mode has anything to do with this error message.
--Barry Finkel
On 3/18/19 6:44 PM, Jim Dory wrote:
Does your outgoing MTA add an X-Ham-Report: header? Perhaps it does so in a non-compliant way.
This one you can't do much about. The recipient thinks it's spammy, but you'll never know why without a lot of trial and error.
If your problems are mostly with digests, you can subscribe yourself (at alternate addresses) to both MIME and plain digests and then when you're looking at a bounce, at least you'll have your copy of the message that was sent for analysis.
Good thing you asked, although if you set digestable to No, at that point and on every web admin access after you will get a big, bold
Warning: You have digest members, but digests are turned off. Those people will not receive mail. Affected member(s) [list of digest members]
You will need to set those members to non-digest either manually or via a script like <https://www.msapiro.net/scripts/set_nodigest.py>
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Mar 18, 2019 at 6:19 PM Mark Sapiro <mark@msapiro.net> wrote:
Thanks very much Mark and Barry. I'll look into the errors further on my own. Don't know that much about this stuff.. I'm on a VPS with root access.. it uses Exim and Apache SpamAssassin. I can log in via WHM or CPanel. So I'll poke around. Things were pretty quiet for awhile, but our VPS was just migrated to another server by our host, so I think a few things may have gotten changed.
As for disabling Digest, I have a fair number of users that would probably be upset. Just tonight at a social event in town, one user mentioned our mailing list (comparing it to a local facebook group) saying how he enjoyed having the digest so it didn't ping his phone for every message rolling in. (Some of us know how to filter that, but most users may not). So I probably will not disable it.
/jim
Jim Dory writes:
On Mon, Mar 18, 2019 at 4:39 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
(You probably should also disable the Digestable option in [Digest options]
When I wrote this, I was thinking of a new list, in the context of personalization as a hack to limit the number of recipients per session. The point was to prevent users from setting their subscriptions to digest. As far as I know, there is no personalization of digest messages, so opting for digest would (theoretically) defeat personalization used to force one message per session operation. I don't know of any reason why digests would increase the probablity of either a bounce or being considered spam in and of themselves.
[Would disabling digestable] prevent issues like this?
As others mentioned, it won't affect those specific issues. Both relate to the content of the post, not the rate or mode of multiple deliveries to the same host. Of course if the digest contains other posts, the recipients lose those too, but that's a choice we prefer to leave up to the subscriber herself.
That's a decision for the administrator. Our rationale for defaulting to no mail in this case is that people who select digest have explicitly opted for a small number of messages, and would likely be more annoyed by an unexpectedly large number of messages than by missing messages (which they can read in the archives in most cases). YMMV on that judgment -- of course, I suppose many would expect automatic switching to non-digest mode as you did -- but that's our rationale for a default to non-delivery.
On 3/18/2019 4:51 AM, Davide Marchi wrote:
This is what VERP deliveries are for. The FROM address is tagged so that bounces can be clearly identified.
https://wiki.list.org/DOC/So%20what%20is%20this%20VERP%20stuff
david
-- IBM i on Power Systems: For when you can't afford to be out of business!
I'm riding 615 miles (Yes, you read that right) in the American Diabetes Association's Tour de Cure to raise money for diabetes research, education, advocacy, and awareness. You can make a tax-deductible donation to my ride by visiting https://mideml.diabetessucks.net.
You can see where my donations come from by visiting my interactive donation map ... https://mideml.diabetessucks.net/map (it's a geeky thing).
I may have diabetes, but diabetes doesn't have me!
Davide Marchi writes:
That depends on what you mean by "monitor" and "delivery errors". If you mean mail refused by some system (typically the final recipient) and returned to Mailman, that is what we call "bounces". Mailman can be configured on the [Bounce processing] screen to send mail to the list owner (only, not moderators or the site administrator) four types of notifications about bounces. The default is usually appropriate.
In some cases recipient systems don't identify the bounced recipient. In that case you can use VERP as described below, which arranges that Mailman receives all bounces in such a way that the recipient can be identified (a special-purpose return address is allocated per recipient).
There is no web-based monitor in Mailman 2. These notifications are sent by email to the list owner. However, if your email provider uses a web-based manager like cPanel, you may have some access to this information (I don't use cPanel, so I don't know what features it provides for monitoring system health).
If you mean any other kind of delivery failure that is detected by the Mailman host, but is not returned to Mailman, then it is simply impossible for Mailman to even be aware of them, and you would have to check the MTA (mail server) logs. And of course there are delivery errors that happen and never are returned to the Mailman host at all (you find out about these because subscribers tell you). There's nothing that can be done about those by Mailman, either.
You should set DMARC Moderation Action to Munge From in the [Privacy options -- Sender filters] screen. (Strictly speaking this isn't spam filtering, it's mitigating breakage induced by incompetent email providers.) And of course you should filter incoming mail for spam (and if you have any human users besides root on the Mailman host, outgoing mail too -- this is why it's a good idea to have a dedicated host for Mailman!) The MTA should be set up to use DKIM and SPF.
increasing the number of sending sessions/decreasing individual emails sent per session?
It's not obvious to me that this is very useful: most of the big providers count the number of times a message is received by any method. They don't care if you do it one at a time (although they may shut you down before any mail is delivered if they are counting the recipients per session). The usual reason for throttling is because Mailman's provider doesn't like accepting huge numbers of addressees from Mailman, not because you have tons of subscribers at one provider.
Only very crudely, in a site-wide way or very drastically per-list. The site-wide option works with Mailman's internal "SMTPDirect" DELIVERY_MODULE (which is the default). Set SMTP_MAX_RCPTS to some "reasonable" figure in mm_cfg.py. This applies to all outgoing connections.
Alternatively, you can set VERP_INTERVAL=1 in mm_cfg.py, which ensures that every message is a one-subscriber-per-SMTP-session sitewide, or for each list you can enable Personalization in the [Non-digest options] which has the same effect for one list. (You probably should also disable the Digestable option in [Digest options].
Steve
On 3/18/19 5:39 AM, Stephen J. Turnbull wrote:
However errors that occur during SMTP from Mailman to the outgoing MTA are returned to Mailman. Mailman sees the SMTP status return and if it is a 5xx status, scores that as a bounce and if it is a 4xx status retries sending the message to the failed recipient(s) at configurable intervals for a configurable time.
The retry interval is DELIVERY_RETRY_WAIT with a default of one hour, but in MM 2.1.23, DELIVERY_RETRY_WAIT is ignored and the interval is 15 minutes.
The duration of retries is DELIVERY_RETRY_PERIOD with a default of 5 days, after which Mailman gives up without scoring a bounce.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Mar 18, 2019 at 4:39 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
(You probably should also disable the Digestable option in [Digest options]
I'm also receiving several bounces back for errors such as: "SMTP error from remote mail server after end of data: 554 Failed: Malformed MIME field: X-Ham-Report:"
Or: "SMTP error from remote mail server after end of data: 554 5.7.1 [P4] Message blocked due to spam content in the message."
Mostly associated with the digest. Curious about the statement above from Steve: is disabling the digest option something people do regularly to prevent issues like this? and if so - if I were to disable it now, what happens to those members who are using that option? I assume they just start automatically receiving non-digest messages, but figured I should ask.
thanks, JD
On 3/18/2019 8:44 PM, Jim Dory wrote:
The second "error" message is one over which you have little control. The recipient's mail system detected content in the digest that it considered spam. I assume that if the messages in the digest had been sent individually in non-digest mode, then the message that had the objectionable content would have been similarly flagged.
As for the first "error" - A quick Google search showed that the
X-Ham-Report:header line is written by Cpanel when it detects that the message is not spam. I do not know if any other software writes this line. I am not an expert in MIME, but I do not see how any "X-" header line could be treated as a malformed MIME header.
If you have users who are using digest mode, then removing that option will cause consternation with those users. And I doubt that the digest mode has anything to do with this error message.
--Barry Finkel
On 3/18/19 6:44 PM, Jim Dory wrote:
Does your outgoing MTA add an X-Ham-Report: header? Perhaps it does so in a non-compliant way.
This one you can't do much about. The recipient thinks it's spammy, but you'll never know why without a lot of trial and error.
If your problems are mostly with digests, you can subscribe yourself (at alternate addresses) to both MIME and plain digests and then when you're looking at a bounce, at least you'll have your copy of the message that was sent for analysis.
Good thing you asked, although if you set digestable to No, at that point and on every web admin access after you will get a big, bold
Warning: You have digest members, but digests are turned off. Those people will not receive mail. Affected member(s) [list of digest members]
You will need to set those members to non-digest either manually or via a script like <https://www.msapiro.net/scripts/set_nodigest.py>
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Mar 18, 2019 at 6:19 PM Mark Sapiro <mark@msapiro.net> wrote:
Thanks very much Mark and Barry. I'll look into the errors further on my own. Don't know that much about this stuff.. I'm on a VPS with root access.. it uses Exim and Apache SpamAssassin. I can log in via WHM or CPanel. So I'll poke around. Things were pretty quiet for awhile, but our VPS was just migrated to another server by our host, so I think a few things may have gotten changed.
As for disabling Digest, I have a fair number of users that would probably be upset. Just tonight at a social event in town, one user mentioned our mailing list (comparing it to a local facebook group) saying how he enjoyed having the digest so it didn't ping his phone for every message rolling in. (Some of us know how to filter that, but most users may not). So I probably will not disable it.
/jim
Jim Dory writes:
On Mon, Mar 18, 2019 at 4:39 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
(You probably should also disable the Digestable option in [Digest options]
When I wrote this, I was thinking of a new list, in the context of personalization as a hack to limit the number of recipients per session. The point was to prevent users from setting their subscriptions to digest. As far as I know, there is no personalization of digest messages, so opting for digest would (theoretically) defeat personalization used to force one message per session operation. I don't know of any reason why digests would increase the probablity of either a bounce or being considered spam in and of themselves.
[Would disabling digestable] prevent issues like this?
As others mentioned, it won't affect those specific issues. Both relate to the content of the post, not the rate or mode of multiple deliveries to the same host. Of course if the digest contains other posts, the recipients lose those too, but that's a choice we prefer to leave up to the subscriber herself.
That's a decision for the administrator. Our rationale for defaulting to no mail in this case is that people who select digest have explicitly opted for a small number of messages, and would likely be more annoyed by an unexpectedly large number of messages than by missing messages (which they can read in the archives in most cases). YMMV on that judgment -- of course, I suppose many would expect automatic switching to non-digest mode as you did -- but that's our rationale for a default to non-delivery.
participants (6)
-
Barry S. Finkel -
David Gibbs -
Davide Marchi -
Jim Dory -
Mark Sapiro -
Stephen J. Turnbull