Excessive or fatal bounces issue
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello, I've been having some issues lately. We are using Mailman 2.1.15. I've looked over the FAQ but the only thing that I could find is to contact the ISPs and ask to be whitelisted - am I understanding this correctly? Here is a brief description of the issue:
Our server hosts 5 listserv, the biggest one counts over 530 subscribed members from 55 different countries.
We had received sporadic bounces triggered by emails sent to these lists since the install of the server in 2016, but in the last months the number of bounces we get has increased. We receive these bounce notifications informing us that the given *subscription has been disabled* due to *excessive or fatal bounces *(up to around 30 bounce notifications triggered by a single email sent to one of the listservs). Every time we get such notifications, we untick the bounce under “nomail [reason]" in the admin management site for each of the email addresses that haven been disabled by the system.
Furthermore, it has happened several times in the past few months that a number of email addresses have been automatically removed from the listservs (at the same time, from different countries). We have not found an explanation for why this happens or a solution, so the only thing we are able to do is to manually add them back to the listservs.
We have also asked these colleagues for which we have received bounces to check whether there is anything blocking the listservs’ emails from their side (note that many of them work in ministries or agencies with strong security features), and to add the server IP to the white list, but the problem persists.
Thank you for your time,
Florin Pasăre,
![](https://secure.gravatar.com/avatar/0fbcef57d028af495d8c9a5992405f78.jpg?s=120&d=mm&r=g)
On Mon, Jul 31, 2023 at 6:23 PM Florin Pasăre <fpasare@gmail.com> wrote:
In summary:
- Your MTA logs probably have clues - with the reasons for the bounce(s). I'd love to see what the recipient servers think.
- You have not mentioned if you DKIM-sign the list emails and have DMARC mitigation.
- Do you have SPF records published?
- Do you have stats on the bounces?
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello and thank you for your reply. Here are the answers to your questions:
- Your MTA logs probably have clues - with the reasons for the bounce(s). I'd love to see what the recipient servers think.
Here some sample about the bounces. Sorry but for privacy I hid the email name but left the domain
<xxx@cfwb.be>: host seg-02.etnic.be[193.53.34.104] said: 550 #5.7.1
DMARC unauthenticated mail is prohibited. (in reply to end of DATA command)
<xxx@bmbwf.gv.at>: host smtp01.noc-science.at[144.65.134.41] said:
554 5.7.1 This email from IP 185.224.197.180 has been rejected. The
message was detected as spam. (in reply to end of DATA command)
<xxx@ext.mesr.etat.lu>: host incoming-1.mail.etat.lu[185.106.25.64]
said: 550 #5.7.1 DMARC unauthenticated mail is prohibited. (in reply to
end
of DATA command)
<xxx@msmt.cz>: host jarilof.msmt.cz[195.113.76.11] said: 554 5.7.1
This email from IP 185.224.197.180 has been rejected. The email message
was
detected as spam. (in reply to end of DATA command)
<xxx@ufm.dk>: host mx7.sitnet.dk[188.64.157.7] said: 550 #5.7.1 DMARC
unauthenticated mail is prohibited. (in reply to end of DATA command)
<xxx@mp.gov.me>: host
primary.us.email.fireeyecloud.com[3.93.93.44] said: 550 5.7.26 ETP205
DMARC
Failure for domain (<qqi.ie>) -
3yPiBJU-67053-09423C05683CD7BFB463fd19b7b
(in reply to end of DATA command)
<xxx@ufm.dk>: host mx5.sitnet.dk[188.64.157.5] said: 550 #5.7.1 DMARC
unauthenticated mail is prohibited. (in reply to end of DATA command)
<xxx@nzqa.govt.nz>: host nzqasmtp.nzqa.govt.nz[163.7.134.150] said:
550 5.7.1 rejected by DMARC policy for qqi.ie (in reply to end of DATA
command)
1. DO you DKIM-sign the list emails and have DMARC mitigation.
Yes, we do.
- Do you have SPF records published?
Yes, we do.
- Do you have stats on the bounces?
Unfortunately I haven’t a stats but only the mail archive. I don't know if there is a feature in Mailman that lets me see bounce statistics.
We done some test with mailtester.com, below the results.
https://www.mail-tester.com/test-92yqfu796
Thank you,
Florin Pasăre,
On Mon, Jul 31, 2023 at 6:55 PM Odhiambo Washington <odhiambo@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 8/3/23 20:17, Florin Pasăre wrote:
You need to ensure your outgoing mail is DKIM signed by your domain and you need to apply DMARC mitigations. These mitigations include the General Options -> from_is_list setting and the Privacy options... -> Sender filters -> dmarc_moderation_action setting.
The former of these was first implemented in Mailman 2.1.16 and the latter in 2.1.18, although if your 2.1.15 is a vendor package, some of this may have been backported.
Without DMARC mitigations, post from users in domains that publish DMARC policies of reject will probably be bounced by any recipient ISPs that check DMARC.
See the FAQ articles at https://wiki.list.org/x/17891458 and https://wiki.list.org/x/17891477 for more information about DMARC.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/b273ab068bc220d17a3e4c710c401c4b.jpg?s=120&d=mm&r=g)
On 7/31/2023 5:53 AM, Florin Pasăre wrote:
We are using Mailman 2.1.15.
Why are you using such an old version of mailman? It's quite possible that a newer version, along with DKIM/SPF DNS records that Odhiambo mentioned, would clear up at least part of the problems.
Also, please look at which domains are bouncing mail and how often. Can you individually send to those people?
Later,
z!
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Florin Pasăre writes:
Hello, I've been having some issues lately. We are using Mailman 2.1.15.
Mailman 2.1.15 was released in 2012, ie, long before the AOL/Yahoo "contact list leak" fiasco. This means your lists are unable to dealwith anti-phishing and anti-spam measures adopted by many sites (and specifically AOL and Yahoo and their successor operators).
You should upgrade to the most recent Mailman 2 for now, and strongly consider upgrade to Mailman 3 in the near future. (Both Mailman 2 and Python 2 are long since "end of life", and increasingly vulnerable to attack as they are not being actively developed.) Make sure you check the upgrade notes for any special procedures and requirements for supporting software.
Recent Mailman provides a large number of important new features. Most important, many recipient sites participate in the DMARC and ARC protocols. The important aspect of DMARC is that a site can declare that emails with "From" address in that domain MUST have a valid DKIM signature from that site (called "From alignment"), otherwise the recipient should reject or discard the email. Because mailing lists frequently make changes such as appending footers to the body and prepending list tags to Subject which invalidate DKIM signatures, From alignment will fail on all mailing lists messages. I cannot be sure without more information whether your users are running into this problem, but it's usually pretty obvious even to non-experts, because mail from a person at site X starts bouncing for everybody at several sites.
Recent Mailman provides two options to deal with this. The first is to rewrite the address in From from 'John Doe <john.doe@example.com>' to something like 'John Doe via Your List <your-list@your-site.org>' for all posts passing through the list. The second is to do the same rewriting, but only for sending domains which publish a "please reject" policy. These are both per-list options.
It also provides a number of other security improvement, in particular protection against cross-site scripting.
The ARC protocol deals with the From alignment problem in a different way. Each participating mail gateway validates the DKIM signature, adds information about the result to the mail, and digitally signs that information. So this basically amounts to each gateway testifying that "everything was OK when it got to me, I made some changes and signed them, check that." Of course this depends on the final recipient trusting intermediary who makes changes, but in most cases that's only your list.
Mailman 2 doesn't provide ARC, but ARC is best implemented in the MTA. Mailman 3 does have an implementation, but we still recommend doing it in the border MTAs as defined by the ARC RFC.
Note that the two DMARC "From" modifications and ARC are all substitutes for each other, with differing advantages and disadvantages. You can get both DMARC features just by upgrading Mailman 2, which you should do anyway to improve the overall security of the Internet. To get ARC, you need to install additional software to your MTAs (or upgrade all the way to Mailman 3).
[Disclaimer: "Listserv" is the trademark of a commercial mailing list manager owned by L-Soft, and should not be used to refer to Mailman installations.]
I'm not sure why you would experience an uptick more recently than say 2014-2016 when many mail servers adopted stricter policies toward malicious mail.
If these are people who are regularly reading their email, this is a somewhat dangerous policy. They're just going to get disabled again, and perhaps unsubscribed. In the meantime, they're losing mail. You really need to investigate why these bounces are happening. By far most delivery failures these days are due to recipient site policy, not equipment or software failures. Occasionally inattentive users exceed mailbox allocations, but that's not so common in these days of freemail providers with default 10GB allocations.
What more can you do?
First, get list owners to check for "Delivery Status Notifications" (DSNs). These often explain in more or less plain language why the mail was rejected. Often it's because it was considered spam or phishing (the less plain language is "administratively prohibited"). In that case it's very likely your posts will get caught frequently.
Second check both the Mailman logs (usually in some directory like /var/lib/mailman/logs) and the MTA logs. MTA logs are usually in directories like /var/log/$MTA_NAME, but also may be in /var/log/mail.log or even the syslog.
If the problem is not obvious, don't hesitate to ask us about them. However, when you do, please provide us with all the logs and DSNs pertaining to a particular rejected message. Do provide the entire log message, including timestamp. You probably should post such requests to *this list*, because the developers are not necessarily the best people to interpret them, and we are a small group. Often you will get the best advice from other users who have had similar experiences.
Please check any materials you forward to this list for user logins and passwords and redact them. (There shouldn't be any, but ....)
If you are concerned about revealing information about your internal network configuration, feel free to redact domain names, IP addresses, and email addresses. However, they should be recognizable as what they are (use "example.com", "10.1.2.3", and "john.doe@example.com", not "REDACTED"), Also, each unique entity should received a unique substitution consistently. For domains, "example.com", "example.org", and "example.net" (at least, there may be others) are "black hole" sites registered by the IETF (I think) for use in documentation, so they won't be confused with real domains. Similarly, I recommend IPs be substituted with dotted-quads starting with "10.", because those IPs are defined to be private and unreachable from the public Internet.
Alternatively, if you need to explain sensitive details about your systems you can write to me and Mark Sapiro personally.
This is part of a normal process. When Mailman receives a bounce, it increments the bounce count for that address. If the bounce count is lower than the disable level, it will continue to try to send messages to that address (and no more bounces are counted for 24 hours). Once it reaches the disable level, the subscription is disabled. If Mailman continues to receive bounces for that address, eventually the address will be unsubscribed.
List owners can tune this process by setting the number of bounces for the disable and unsubscribe levels, as well as the number of days with no bounces until the bounce count gets set to zero.
There are several reasons why Mailman might continue to send list- related traffic to a disabled address, but this reflex of reenabling disabled addresses without solving the underlying problem is definitely one.
Of course, there's not much else you can do if you're not an expert. But please, in the future, when your own systems are denying service to your users, *do* come talk to us (or whoever the vendor is) as soon as you realized it's a recurring problem.
Also, please note that Mailman distinguishes between hard bounces, where some server said "we will never accept this message for this address", and soft bounces, where the server said "we can't accept this message now but please try again later". The MTA (usually) or sometimes Mailman does exactly that: it adds the message to a "retry" queue and tries again. Hard bounces, on the other hand, means that the message was never delivered to that subscriber. Your users are losing email.
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello, Stephen,
Thank you for your response. I know we should upgrade, but because of company policies this is not really an option at the moment, at least not an upgrade to Mailman 3. The main issue is fear of losing the email archive. I've replied with some errors/bounces that we received. I will try to reply to your other questions as soon as I get the information.
Thank you,
Florin Pasăre,
On Tue, Aug 1, 2023 at 1:48 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Florin Pasăre writes:
Thank you for your response. I know we should upgrade, but because of company policies this is not really an option at the moment,
Like Carl and Mark, I strongly recommend upgrading to most recent Mailman 2. This is straightforward, and during the process you can get round-the-clock support here because everyone has done it several times. You don't need to get the attention of developers, community support is excellent.
You do not have to worry about losing archives from a Mailman 2 upgrade, because the archival format is just the mbox that your MTA produces for you. The web pages are a presentation that can be rebuilt (at some cost in time) at any time. In fact, the most recent month's pages are rebuilt from mboxes every day by a cron job. Even the URLs of individual messages remain the same (unless you edit the mbox by removing or reordering messages). (If you use an alternative to the bundled "Pipermail" archiver, you still probably have all the mboxes somewhere in the system, but you should check that.)
at least not an upgrade to Mailman 3.
That is a project that will require some planning in any case. You can preserve the mbox files (and even leave the old Mailman 2 archive CGIs running), and in that sense you won't lose archives. However, the version of the Python email package in Python 3 does not cope well with some of the weird things you find in Mailman 2, and we have not yet learned how to catch and repair all of them at import time. Also, importing messages into HyperKitty changes all the message URLs (completely different format based on hashed Message-IDs rather than serial number). So you probably should think in terms of a man-week for planning, and a man-week or so for execution, anyway.
I recommend you start "softening up" the CIO for a migration in a year or two. Eventually it's going to become painful to maintain Python 2 applications because even the oldest LTS distributions don't provide old system libraries needed by Python 2, or because QA or CISO says you can't use them any more because they're too old/vulnerable.
Steve
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello and thank you. I will forward these replies to the dev that manages the servers and see if we can get an upgrade to the latest version of Mailman 2 and continue from there. Thank you again for your time.
Florin Pasăre,
On Fri, Aug 4, 2023 at 10:26 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello everyone,
After a long time, we've finally managed to upgrade to the latest version of Mailman 2. Some of the original issues are resolved, thank you, but now we have different issues, more details below:
I would like to ask you where the system gets the names you see in the "sent by" field, because sometimes the name doesn't make it clear who is sending the email and we would like to know if it is something we can control or change.
Another question is where the system gets the name of the person who appears in the "reply to" field.
I've already seen wrong/different than what was expected names twice using Mail on Macbook.
The message is sent, for example, by Mario Rossi, and the system adds his email address to the cc. The problem is the reply-to field, because there one sees the name of, for example, Claudio Bianchi, which has nothing to do with this message and so I don't understand why his name appears above the general address of the mailing list.
With MS Outlook instead what one sees in the sent-by field is the following: list name list-bounces@domain.ext on behalf of XXX via list name list@domain.ext.
Is it possible to have list@domain.ext instead of list-bounces@domain.ext?
If you have any hints for where to look or any ideas, it would be great.
Thank you,
Florin Pasăre,
On Fri, Aug 4, 2023 at 4:50 PM Florin Pasăre <fpasare@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 11/22/23 4:03 AM, Florin Pasăre wrote:
There are lots of possibilities here depending on your list's settings and the MUA you use to read the mail.
If you are applying DMARC Munge From mitigations to the mail, a message
which is From: Joe User <user@example.com>
sent to list@example.net
will have it's From: changed to From: Joe User via Listname <list@example.net>
. If there is no display name in the original From:
and user@example.com is a list member, that member's name from the list
membership is used unless the member has no name entry. Otherwise, the
local part of the email address is used as in From: user--- via Listname <list@example.net>
. See
https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/...
for details.
In this case, the original From: is added to Cc: or Reply-To: depending on list settings. See https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/... for details.
If the list is not applying DMARC Munge From mitigations to the mail the From: is not changed and nothing is added to Cc: or Reply-To:.
Other things might be done to Reply-To: depending on the list's General Options first_strip_reply_to, reply_goes_to_list and reply_to_address settings.
I've already seen wrong/different than what was expected names twice using Mail on Macbook.
Various MUAs have their own rules for what they display as the message sender which may not be the literal value in From:.
See https://wiki.list.org/x/4030534 particularly the note at the bottom of the page.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Florin Pasăre writes:
I would like to ask you where the system
What do you mean by "the system"? Mailman? The mail servers (these are the programs that exchange mail between hosts on the Internet)? Your mail client (the program you use to read and send mail)? "Something" between the author's fingers and your eyes?
gets the names you see in the "sent by" field,
What "sent by" field? Do you mean something you see in your mail client? Maybe the From field? (There is no standard "sent by" field for email,)
The mail standard RFC 5322 specifies a "From" field which contains the address of the author and optionally a display name for them. This is set by the author's mail client, and according to the standard should not be changed by anyone else. However, (1) mail clients such as Outlook and Apple Mail frequently change, interpret, or add random stuff to the From field when they display the author's identity, and (2) there is an anti-phishing protocol called DMARC according to which a sending system can request rejection of mail claiming to be from that system unless a valid digital signature is present. Since some things that a mailing list does (like adding unsubscription information as legally required in many places) break the signature, Mailman has an option to change the From field from "Some User <user@domain.tld>" to "Some User via This List <list@listhost.tld>". This DMARC mitigation is only used if the list owner explicitly configures it.
Another question is where the system gets the name of the person who appears in the "reply to" field.
According to RFC 5322, that field also should only be set by the author of the original message. However, nonconforming behavior is common, and under certain circumstances Mailman will set that field:
- By default, Mailman will not touch that field, but will pass it through if it exists.
- When configured to do so, Mailman will add a fixed address (usually the list's posting address) that the list owner specifies to Reply-To.
- Alternatively, when Mailman is configured to change the From address to mitigate rejections at subscribers' systems due to the DMARC protocol, Mailman will add the original From address to Reply-To.
I think in the case of setting Reply-To to an specific address, the list owner may specifically ask Mailman to remove other addresses. I don't think that any mail servers (such as Postfix or Exchange) ever touch Reply-To.
I don't recall for sure if there are circumstances where Mailman changes Cc, but I'm pretty sure it never does. I don't think that any mail servers (such as Postfix or Exchange) ever touch Cc, only the originating mail client does.
What do you mean by "above"? Please quote exactly what is on screen. (Sorry, this list doesn't permit screenshots.)
By "sent-by field" do you mean the "From" field?
That sort of looks like something Mailman might produce, but not exactly (which means Mailman did *not* produce it). I believe in the case of DMARC reformatting Mailman produces the "XXX via list name <list@domain.ext>" portion, but it will *definitely* not put "list-bounces" anywhere in From. Check that you are quoting exactly.
If so, I think that is Outlook hallucinating. Outlook definitely displays garbage under some conditions, and I think I remember this particular problem being an Outlook misfeature. Specifically, it grabs the content of the "Sender" field and puts it before "on behalf of".
Is it possible to have list@domain.ext instead of list-bounces@domain.ext?
The <list-bounces@domain.ext> is the address Mailman puts in the "Sender" field (normally not displayed by mail clients). If Sender is present in the mail header, mail servers send various administrative messages to Sender rather than From or Reply-To, including non-delivery messages. You do not want to change the "Sender" field to the list-post address, because all those administrative messages would then go to the list, and many would likely be distributed to the subscribers. You do not want to remove the "Sender" field, because without it such messages would go to post authors but they probably cannot do anything useful with them (and sometimes there are many for just one post).
If you have any hints for where to look or any ideas, it would be great.
My suspicion is that you are stuck: the problem is that the mail clients you are using do a very bad job of presenting header information. I'm not going to tell you to change clients, but I can't think of anything you can change in Mailman that would make it work better with Outlook or Apple Mail.
If you want to see if there is *anything* we can do:
Start by quoting exactly what you see on the screen, and continue to specify the mail client used as you did in this message. The problem is that the mail standard provides a minimum of 6 different ways to specify who sent a message, each with well-defined, unique meaning to mail programs. None of them are called "sent-by", and "good" mail clients report the header fields using the technical names precisely to make conversations like this one go more smoothly. An exact quote of the full header information as presented by the mail client will help us guess more accurately what is going on.
Do not say "the system does X" unless you (1) identify which program is doing X and (2) you are quite sure that that program is the one that is doing X. If X is what you see on screen, say that, and what program you are using to view that content.
If possible, use an open-source mail client such as ThunderBird to view the message and report what you see there. Outlook and Apple Mail both present the message in ways that make it difficult to understand what the message header actually contains.
Regards, Steve
![](https://secure.gravatar.com/avatar/0fbcef57d028af495d8c9a5992405f78.jpg?s=120&d=mm&r=g)
On Mon, Jul 31, 2023 at 6:23 PM Florin Pasăre <fpasare@gmail.com> wrote:
In summary:
- Your MTA logs probably have clues - with the reasons for the bounce(s). I'd love to see what the recipient servers think.
- You have not mentioned if you DKIM-sign the list emails and have DMARC mitigation.
- Do you have SPF records published?
- Do you have stats on the bounces?
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello and thank you for your reply. Here are the answers to your questions:
- Your MTA logs probably have clues - with the reasons for the bounce(s). I'd love to see what the recipient servers think.
Here some sample about the bounces. Sorry but for privacy I hid the email name but left the domain
<xxx@cfwb.be>: host seg-02.etnic.be[193.53.34.104] said: 550 #5.7.1
DMARC unauthenticated mail is prohibited. (in reply to end of DATA command)
<xxx@bmbwf.gv.at>: host smtp01.noc-science.at[144.65.134.41] said:
554 5.7.1 This email from IP 185.224.197.180 has been rejected. The
message was detected as spam. (in reply to end of DATA command)
<xxx@ext.mesr.etat.lu>: host incoming-1.mail.etat.lu[185.106.25.64]
said: 550 #5.7.1 DMARC unauthenticated mail is prohibited. (in reply to
end
of DATA command)
<xxx@msmt.cz>: host jarilof.msmt.cz[195.113.76.11] said: 554 5.7.1
This email from IP 185.224.197.180 has been rejected. The email message
was
detected as spam. (in reply to end of DATA command)
<xxx@ufm.dk>: host mx7.sitnet.dk[188.64.157.7] said: 550 #5.7.1 DMARC
unauthenticated mail is prohibited. (in reply to end of DATA command)
<xxx@mp.gov.me>: host
primary.us.email.fireeyecloud.com[3.93.93.44] said: 550 5.7.26 ETP205
DMARC
Failure for domain (<qqi.ie>) -
3yPiBJU-67053-09423C05683CD7BFB463fd19b7b
(in reply to end of DATA command)
<xxx@ufm.dk>: host mx5.sitnet.dk[188.64.157.5] said: 550 #5.7.1 DMARC
unauthenticated mail is prohibited. (in reply to end of DATA command)
<xxx@nzqa.govt.nz>: host nzqasmtp.nzqa.govt.nz[163.7.134.150] said:
550 5.7.1 rejected by DMARC policy for qqi.ie (in reply to end of DATA
command)
1. DO you DKIM-sign the list emails and have DMARC mitigation.
Yes, we do.
- Do you have SPF records published?
Yes, we do.
- Do you have stats on the bounces?
Unfortunately I haven’t a stats but only the mail archive. I don't know if there is a feature in Mailman that lets me see bounce statistics.
We done some test with mailtester.com, below the results.
https://www.mail-tester.com/test-92yqfu796
Thank you,
Florin Pasăre,
On Mon, Jul 31, 2023 at 6:55 PM Odhiambo Washington <odhiambo@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 8/3/23 20:17, Florin Pasăre wrote:
You need to ensure your outgoing mail is DKIM signed by your domain and you need to apply DMARC mitigations. These mitigations include the General Options -> from_is_list setting and the Privacy options... -> Sender filters -> dmarc_moderation_action setting.
The former of these was first implemented in Mailman 2.1.16 and the latter in 2.1.18, although if your 2.1.15 is a vendor package, some of this may have been backported.
Without DMARC mitigations, post from users in domains that publish DMARC policies of reject will probably be bounced by any recipient ISPs that check DMARC.
See the FAQ articles at https://wiki.list.org/x/17891458 and https://wiki.list.org/x/17891477 for more information about DMARC.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/b273ab068bc220d17a3e4c710c401c4b.jpg?s=120&d=mm&r=g)
On 7/31/2023 5:53 AM, Florin Pasăre wrote:
We are using Mailman 2.1.15.
Why are you using such an old version of mailman? It's quite possible that a newer version, along with DKIM/SPF DNS records that Odhiambo mentioned, would clear up at least part of the problems.
Also, please look at which domains are bouncing mail and how often. Can you individually send to those people?
Later,
z!
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Florin Pasăre writes:
Hello, I've been having some issues lately. We are using Mailman 2.1.15.
Mailman 2.1.15 was released in 2012, ie, long before the AOL/Yahoo "contact list leak" fiasco. This means your lists are unable to dealwith anti-phishing and anti-spam measures adopted by many sites (and specifically AOL and Yahoo and their successor operators).
You should upgrade to the most recent Mailman 2 for now, and strongly consider upgrade to Mailman 3 in the near future. (Both Mailman 2 and Python 2 are long since "end of life", and increasingly vulnerable to attack as they are not being actively developed.) Make sure you check the upgrade notes for any special procedures and requirements for supporting software.
Recent Mailman provides a large number of important new features. Most important, many recipient sites participate in the DMARC and ARC protocols. The important aspect of DMARC is that a site can declare that emails with "From" address in that domain MUST have a valid DKIM signature from that site (called "From alignment"), otherwise the recipient should reject or discard the email. Because mailing lists frequently make changes such as appending footers to the body and prepending list tags to Subject which invalidate DKIM signatures, From alignment will fail on all mailing lists messages. I cannot be sure without more information whether your users are running into this problem, but it's usually pretty obvious even to non-experts, because mail from a person at site X starts bouncing for everybody at several sites.
Recent Mailman provides two options to deal with this. The first is to rewrite the address in From from 'John Doe <john.doe@example.com>' to something like 'John Doe via Your List <your-list@your-site.org>' for all posts passing through the list. The second is to do the same rewriting, but only for sending domains which publish a "please reject" policy. These are both per-list options.
It also provides a number of other security improvement, in particular protection against cross-site scripting.
The ARC protocol deals with the From alignment problem in a different way. Each participating mail gateway validates the DKIM signature, adds information about the result to the mail, and digitally signs that information. So this basically amounts to each gateway testifying that "everything was OK when it got to me, I made some changes and signed them, check that." Of course this depends on the final recipient trusting intermediary who makes changes, but in most cases that's only your list.
Mailman 2 doesn't provide ARC, but ARC is best implemented in the MTA. Mailman 3 does have an implementation, but we still recommend doing it in the border MTAs as defined by the ARC RFC.
Note that the two DMARC "From" modifications and ARC are all substitutes for each other, with differing advantages and disadvantages. You can get both DMARC features just by upgrading Mailman 2, which you should do anyway to improve the overall security of the Internet. To get ARC, you need to install additional software to your MTAs (or upgrade all the way to Mailman 3).
[Disclaimer: "Listserv" is the trademark of a commercial mailing list manager owned by L-Soft, and should not be used to refer to Mailman installations.]
I'm not sure why you would experience an uptick more recently than say 2014-2016 when many mail servers adopted stricter policies toward malicious mail.
If these are people who are regularly reading their email, this is a somewhat dangerous policy. They're just going to get disabled again, and perhaps unsubscribed. In the meantime, they're losing mail. You really need to investigate why these bounces are happening. By far most delivery failures these days are due to recipient site policy, not equipment or software failures. Occasionally inattentive users exceed mailbox allocations, but that's not so common in these days of freemail providers with default 10GB allocations.
What more can you do?
First, get list owners to check for "Delivery Status Notifications" (DSNs). These often explain in more or less plain language why the mail was rejected. Often it's because it was considered spam or phishing (the less plain language is "administratively prohibited"). In that case it's very likely your posts will get caught frequently.
Second check both the Mailman logs (usually in some directory like /var/lib/mailman/logs) and the MTA logs. MTA logs are usually in directories like /var/log/$MTA_NAME, but also may be in /var/log/mail.log or even the syslog.
If the problem is not obvious, don't hesitate to ask us about them. However, when you do, please provide us with all the logs and DSNs pertaining to a particular rejected message. Do provide the entire log message, including timestamp. You probably should post such requests to *this list*, because the developers are not necessarily the best people to interpret them, and we are a small group. Often you will get the best advice from other users who have had similar experiences.
Please check any materials you forward to this list for user logins and passwords and redact them. (There shouldn't be any, but ....)
If you are concerned about revealing information about your internal network configuration, feel free to redact domain names, IP addresses, and email addresses. However, they should be recognizable as what they are (use "example.com", "10.1.2.3", and "john.doe@example.com", not "REDACTED"), Also, each unique entity should received a unique substitution consistently. For domains, "example.com", "example.org", and "example.net" (at least, there may be others) are "black hole" sites registered by the IETF (I think) for use in documentation, so they won't be confused with real domains. Similarly, I recommend IPs be substituted with dotted-quads starting with "10.", because those IPs are defined to be private and unreachable from the public Internet.
Alternatively, if you need to explain sensitive details about your systems you can write to me and Mark Sapiro personally.
This is part of a normal process. When Mailman receives a bounce, it increments the bounce count for that address. If the bounce count is lower than the disable level, it will continue to try to send messages to that address (and no more bounces are counted for 24 hours). Once it reaches the disable level, the subscription is disabled. If Mailman continues to receive bounces for that address, eventually the address will be unsubscribed.
List owners can tune this process by setting the number of bounces for the disable and unsubscribe levels, as well as the number of days with no bounces until the bounce count gets set to zero.
There are several reasons why Mailman might continue to send list- related traffic to a disabled address, but this reflex of reenabling disabled addresses without solving the underlying problem is definitely one.
Of course, there's not much else you can do if you're not an expert. But please, in the future, when your own systems are denying service to your users, *do* come talk to us (or whoever the vendor is) as soon as you realized it's a recurring problem.
Also, please note that Mailman distinguishes between hard bounces, where some server said "we will never accept this message for this address", and soft bounces, where the server said "we can't accept this message now but please try again later". The MTA (usually) or sometimes Mailman does exactly that: it adds the message to a "retry" queue and tries again. Hard bounces, on the other hand, means that the message was never delivered to that subscriber. Your users are losing email.
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello, Stephen,
Thank you for your response. I know we should upgrade, but because of company policies this is not really an option at the moment, at least not an upgrade to Mailman 3. The main issue is fear of losing the email archive. I've replied with some errors/bounces that we received. I will try to reply to your other questions as soon as I get the information.
Thank you,
Florin Pasăre,
On Tue, Aug 1, 2023 at 1:48 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Florin Pasăre writes:
Thank you for your response. I know we should upgrade, but because of company policies this is not really an option at the moment,
Like Carl and Mark, I strongly recommend upgrading to most recent Mailman 2. This is straightforward, and during the process you can get round-the-clock support here because everyone has done it several times. You don't need to get the attention of developers, community support is excellent.
You do not have to worry about losing archives from a Mailman 2 upgrade, because the archival format is just the mbox that your MTA produces for you. The web pages are a presentation that can be rebuilt (at some cost in time) at any time. In fact, the most recent month's pages are rebuilt from mboxes every day by a cron job. Even the URLs of individual messages remain the same (unless you edit the mbox by removing or reordering messages). (If you use an alternative to the bundled "Pipermail" archiver, you still probably have all the mboxes somewhere in the system, but you should check that.)
at least not an upgrade to Mailman 3.
That is a project that will require some planning in any case. You can preserve the mbox files (and even leave the old Mailman 2 archive CGIs running), and in that sense you won't lose archives. However, the version of the Python email package in Python 3 does not cope well with some of the weird things you find in Mailman 2, and we have not yet learned how to catch and repair all of them at import time. Also, importing messages into HyperKitty changes all the message URLs (completely different format based on hashed Message-IDs rather than serial number). So you probably should think in terms of a man-week for planning, and a man-week or so for execution, anyway.
I recommend you start "softening up" the CIO for a migration in a year or two. Eventually it's going to become painful to maintain Python 2 applications because even the oldest LTS distributions don't provide old system libraries needed by Python 2, or because QA or CISO says you can't use them any more because they're too old/vulnerable.
Steve
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello and thank you. I will forward these replies to the dev that manages the servers and see if we can get an upgrade to the latest version of Mailman 2 and continue from there. Thank you again for your time.
Florin Pasăre,
On Fri, Aug 4, 2023 at 10:26 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
![](https://secure.gravatar.com/avatar/983b8eee387f3ba819cfd60245fd8ba9.jpg?s=120&d=mm&r=g)
Hello everyone,
After a long time, we've finally managed to upgrade to the latest version of Mailman 2. Some of the original issues are resolved, thank you, but now we have different issues, more details below:
I would like to ask you where the system gets the names you see in the "sent by" field, because sometimes the name doesn't make it clear who is sending the email and we would like to know if it is something we can control or change.
Another question is where the system gets the name of the person who appears in the "reply to" field.
I've already seen wrong/different than what was expected names twice using Mail on Macbook.
The message is sent, for example, by Mario Rossi, and the system adds his email address to the cc. The problem is the reply-to field, because there one sees the name of, for example, Claudio Bianchi, which has nothing to do with this message and so I don't understand why his name appears above the general address of the mailing list.
With MS Outlook instead what one sees in the sent-by field is the following: list name list-bounces@domain.ext on behalf of XXX via list name list@domain.ext.
Is it possible to have list@domain.ext instead of list-bounces@domain.ext?
If you have any hints for where to look or any ideas, it would be great.
Thank you,
Florin Pasăre,
On Fri, Aug 4, 2023 at 4:50 PM Florin Pasăre <fpasare@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 11/22/23 4:03 AM, Florin Pasăre wrote:
There are lots of possibilities here depending on your list's settings and the MUA you use to read the mail.
If you are applying DMARC Munge From mitigations to the mail, a message
which is From: Joe User <user@example.com>
sent to list@example.net
will have it's From: changed to From: Joe User via Listname <list@example.net>
. If there is no display name in the original From:
and user@example.com is a list member, that member's name from the list
membership is used unless the member has no name entry. Otherwise, the
local part of the email address is used as in From: user--- via Listname <list@example.net>
. See
https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/...
for details.
In this case, the original From: is added to Cc: or Reply-To: depending on list settings. See https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/... for details.
If the list is not applying DMARC Munge From mitigations to the mail the From: is not changed and nothing is added to Cc: or Reply-To:.
Other things might be done to Reply-To: depending on the list's General Options first_strip_reply_to, reply_goes_to_list and reply_to_address settings.
I've already seen wrong/different than what was expected names twice using Mail on Macbook.
Various MUAs have their own rules for what they display as the message sender which may not be the literal value in From:.
See https://wiki.list.org/x/4030534 particularly the note at the bottom of the page.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Florin Pasăre writes:
I would like to ask you where the system
What do you mean by "the system"? Mailman? The mail servers (these are the programs that exchange mail between hosts on the Internet)? Your mail client (the program you use to read and send mail)? "Something" between the author's fingers and your eyes?
gets the names you see in the "sent by" field,
What "sent by" field? Do you mean something you see in your mail client? Maybe the From field? (There is no standard "sent by" field for email,)
The mail standard RFC 5322 specifies a "From" field which contains the address of the author and optionally a display name for them. This is set by the author's mail client, and according to the standard should not be changed by anyone else. However, (1) mail clients such as Outlook and Apple Mail frequently change, interpret, or add random stuff to the From field when they display the author's identity, and (2) there is an anti-phishing protocol called DMARC according to which a sending system can request rejection of mail claiming to be from that system unless a valid digital signature is present. Since some things that a mailing list does (like adding unsubscription information as legally required in many places) break the signature, Mailman has an option to change the From field from "Some User <user@domain.tld>" to "Some User via This List <list@listhost.tld>". This DMARC mitigation is only used if the list owner explicitly configures it.
Another question is where the system gets the name of the person who appears in the "reply to" field.
According to RFC 5322, that field also should only be set by the author of the original message. However, nonconforming behavior is common, and under certain circumstances Mailman will set that field:
- By default, Mailman will not touch that field, but will pass it through if it exists.
- When configured to do so, Mailman will add a fixed address (usually the list's posting address) that the list owner specifies to Reply-To.
- Alternatively, when Mailman is configured to change the From address to mitigate rejections at subscribers' systems due to the DMARC protocol, Mailman will add the original From address to Reply-To.
I think in the case of setting Reply-To to an specific address, the list owner may specifically ask Mailman to remove other addresses. I don't think that any mail servers (such as Postfix or Exchange) ever touch Reply-To.
I don't recall for sure if there are circumstances where Mailman changes Cc, but I'm pretty sure it never does. I don't think that any mail servers (such as Postfix or Exchange) ever touch Cc, only the originating mail client does.
What do you mean by "above"? Please quote exactly what is on screen. (Sorry, this list doesn't permit screenshots.)
By "sent-by field" do you mean the "From" field?
That sort of looks like something Mailman might produce, but not exactly (which means Mailman did *not* produce it). I believe in the case of DMARC reformatting Mailman produces the "XXX via list name <list@domain.ext>" portion, but it will *definitely* not put "list-bounces" anywhere in From. Check that you are quoting exactly.
If so, I think that is Outlook hallucinating. Outlook definitely displays garbage under some conditions, and I think I remember this particular problem being an Outlook misfeature. Specifically, it grabs the content of the "Sender" field and puts it before "on behalf of".
Is it possible to have list@domain.ext instead of list-bounces@domain.ext?
The <list-bounces@domain.ext> is the address Mailman puts in the "Sender" field (normally not displayed by mail clients). If Sender is present in the mail header, mail servers send various administrative messages to Sender rather than From or Reply-To, including non-delivery messages. You do not want to change the "Sender" field to the list-post address, because all those administrative messages would then go to the list, and many would likely be distributed to the subscribers. You do not want to remove the "Sender" field, because without it such messages would go to post authors but they probably cannot do anything useful with them (and sometimes there are many for just one post).
If you have any hints for where to look or any ideas, it would be great.
My suspicion is that you are stuck: the problem is that the mail clients you are using do a very bad job of presenting header information. I'm not going to tell you to change clients, but I can't think of anything you can change in Mailman that would make it work better with Outlook or Apple Mail.
If you want to see if there is *anything* we can do:
Start by quoting exactly what you see on the screen, and continue to specify the mail client used as you did in this message. The problem is that the mail standard provides a minimum of 6 different ways to specify who sent a message, each with well-defined, unique meaning to mail programs. None of them are called "sent-by", and "good" mail clients report the header fields using the technical names precisely to make conversations like this one go more smoothly. An exact quote of the full header information as presented by the mail client will help us guess more accurately what is going on.
Do not say "the system does X" unless you (1) identify which program is doing X and (2) you are quite sure that that program is the one that is doing X. If X is what you see on screen, say that, and what program you are using to view that content.
If possible, use an open-source mail client such as ThunderBird to view the message and report what you see there. Outlook and Apple Mail both present the message in ways that make it difficult to understand what the message header actually contains.
Regards, Steve
participants (5)
-
Carl Zwanzig
-
Florin Pasăre
-
Mark Sapiro
-
Odhiambo Washington
-
Stephen J. Turnbull