Assistance with altering reply-to behaviours and DMARC
Hi,
First, apologies if this has been discussed before. I run a number of mailman lists on a Centos 6 platform and mailman 2.1.12-25. This version was updated in July as follows: https://rhn.redhat.com/errata/RHSA-2015-1417.html and included fixes for a number of DMARC issues. Including "... With this update, domains with a "reject" DMARC policy are recognized correctly.."
Normally the lists we use are configured with reply_goes_to_list and anonymous_list is off. We have done some testing, and we have not been able to identify settings that will retain reply_goes_to_list functionality and turn on DMARC Mung.
Generally what happens is that the first message is successfully sent to a yahoo.com address and this user can reply to the message as normal. However, mailman adds the original sender to the reply-to meaning that when another user replies to this message reply_goes_to_list doesn't work.
It seemed to me that a small modification to CookHeaders.py could prevent the adding of the original sender email to "Reply-To"
# We also need to put the old From: in Reply-To: in all cases.
if o_from:
add(o_from)
# Set Reply-To: header to point back to this list. Add this lastseemed like a good place to look, but before I try to debug commenting out this didn't prevent the addition of the original sender to Reply-To I would ask for some advice !!
any assistance would be very helpful!! (it maybe that changing the order of addresses in the Reply-To field would all reply_goes_to_list would work?)
thanks Mark.
Mark writes:
Stock Mailman 2.1.12 doesn't do any DMARC detection. This is quite bizarre that they would backport such a feature rather than update to 2.1.18-1 or later. Mailman 2.1 is hardly an unstable package.
You'll have to talk to the vendor who did the modifications.
What do you mean by "doesn't work"? That it doesn't go to the list at all, or that although it goes to the list, it *also* goes (separately) to the original poster?
Not going to the list at all means the MUA is broken IMO; multiple addresses in Reply-To are explicitly allowed by the RFC, and MUAs should reply to all of them.
After disabling the "add(o_from)" operation, if reply_goes_to_list is set, then (1) the reply goes *only* to the list by default, and (2) it is impossible for the user's MUA to automatically format a reply to poster. If reply_goes_to_list is not set, (1) reply by default will go to list (since it's the address in From), and (2) it is impossible to automatically format a reply to poster.
It is our opinion that (2) is a sufficiently important bug that we are unlikely to change this, even to provide an option. But if you want to make that edit, it probably does what you want. Depending on your users, you may or may not get complaints about inability to respond privately.
On 28/08/15 4:37 pm, Stephen J. Turnbull wrote:
I am not sure why the changes were back-ported - maybe it had something to do with the version of Python on the platform. I fully understand that the ability of the developers is limited to supporting known releases, and this release is not a supported one. So I don't expect more than some general guidance :)
I will try to explain, but its only on subsequent replies the issue appears:-
Step 1: email is sent to the list - the mail to DMARC protected domain is received ok by Yahoo web client. Reply-to is set to the list.
Message source of first email is:
From: Mark <mark@incet.com> Reply-To: Test <test@testdomain.org>
Step 2: Reply from Yahoo client sends an email to the list and this has the following source when viewed on Thunderbird MUA
From: Mark H via Test <test@testdomain.org> Reply-To: Mark H <markhnzy@yahoo.com>, Test <test@testdomain.org>
Step 3: A reply from the TB MUA goes to markhnzy@yahoo.com and test@testdomain.org
Because mailman helpfully recognises that a copy has already been sent to the user directly, a copy is not sent to markhnzy@ from the list.
Step 4: Now, when markhnzy@ replies, the conversation is between the recipient and the sender alone.
I agree that this is not a bug as such; its more about the desired behaviour for a specific need of mailman and that new non-RFC compliant behaviour will result.
I tried commenting out if o_from and add(o_from), but original from address was still added to the reply-to(as well as the list address). I did change some permissions and restart mailman and confirm that a new py file was created, so I don't think its that my change wasn't included.
Is there any other code in the standard mailman where the reply-to is populated with the original sender? I realise fully aware that the version of mailman I have here isn't supported, or released, and other bugs may have been introduced.
thanks again for the replies.
Mark.
Mark wrote:
That is interesting, and I wonder if this is something to do with the way that Redhat have implemented the new feature.
On my stock Mailman installation, the headers would look like this:
|From: Mark H via Test <test@testdomain.org> |Reply-To: Test <test@testdomain.org> |CC: Mark H <markhnzy@yahoo.com>
This is a much better imho way of dealing with the issue and keeps the messages flowing through the list. I know if I had the same headers on my list that you are getting, that users would be falling into the same trap, and was something I tested hard before implementing it on my lists.
Is there any way you could just upgrade your installation to the latest 2.1.20? There are some compiling guides out there for Redhat/Centos systems that were designed to get people through this before Redhat backported the feature.
Thanks. Andrew.
Sorry, serious finger fumble there. Please ignore the messed up one.
Andrew Hodgson writes:
That is interesting, and I wonder if this is something to do with the way that Redhat have implemented the new feature.
My guess is that Red Hat backported 2.1.17 or 2.1.18-1. After getting complaints, Mark S fixed things so that the From address was moved to Cc rather than Reply-To in 2.1.19 or 2.1.20.
It's easy enough to do, but beware of installing into the same places that Red Hat/CentOS uses, as that is quite likely to result in various problems if you try to upgrade the Mailman package from the OS vendor. On the other hand, using a source build installed to the default /var/local hierarchy means you'll need to copy the site config mm_cfg.py, the list configs, and the archives to the new location, and teach the webserver and mailserver about the new Mailman. In principle it's pretty straightforward and I haven't hesitated to do so on several occasions, but I can understand why an admin might be nervous (or they might have a rule that all installed software must be controlled by the system's PMS, and that's a good rule that shouldn't be violated lightly).
Mark writes:
In standard Mailman >= 2.1.18-1 (more or less) there are two places. I guess the one you found is the unconditional "from_is_list" setting in CookHeaders.py; sorry, I misled you because I was under them impression that all such changes would take place in CookHeaders. In 2.1.20, there's another, more complex path which involves checking dmarc_moderation_action in SpamDetect.py, deciding which headers to munge in CookHeaders.py:change_header(), and actually doing the work in WrapMessage.py.
I don't have 2.1.18-1 to hand, and I'm not sure that is the version that was backported anyway, so it's hard to be sure what you'd need to change. Probably searching for "dmarc" in SpamDetect and WrapMessage, and "change_header" in CookHeaders, will allow you to figure out how the backport works.
On Fri, Aug 28, 2015 at 01:37:18PM +0900, Stephen J. Turnbull wrote:
That's the way most "stable" Linux distributions do it. I think it has a lot to do with keeping the complex web of interdependencies in check, but probably also has to do with keeping features predictable. Even with security issues, generally, they go to quite a bit of work to backport things, rather than change versions.
Not sure about 6, but the specific way they backported the changes for the DMARC patch on EL5 had to do partially with dependencies... either the fact that the normal version didn't depend on dnspython and this one did, or the version of dnspython required wasn't available in EL5 (looks like they actually ended up vendoring dnspython inside the Mailman package).
Folks who want more up to date versions tend to either jump major releases more often, use third party repos, rebuild RPMs from upstream, or just build from source. We use the vendor provided Mailman package, and for the most part, it works for us, though in cases like this, it is problematic.
w
On 8/28/2015 3:03 AM, Will Yardley wrote:
It appears to me that if someone were to back-port the DMARC changes from 2.1.18 back to 2.1.12, then there is a possibility that there will be problems with the back-port due to code changes between .12 and .18. When I looked at the Ubuntu changes, I saw many changes that were not documented, and I had no idea what they did. Part of the problem in Ubuntu (or any other packager) who wants to create a package for a newer Mailman is that all of these patches, whether created to fix bugs or add features, have to be re-fitted into the code. And that takes time. I am not sure which process is more prone to introduce errors - re-fitting patches into a newer Mailman or trying to re-fit DMARC patches into on older Mailman.
This is why I determined, when I was running a Mailman installation on Ubuntu, that I would create my own package from the SourceForge source instead of installing the Ubuntu Mailman package. It took me a while to figure out how to do it, but once I did it, I had the instructions to re-do it for subsequent SF Mailman releases.
I wanted to know exactly what source I was running, so I could get assistance from the authors via this list.
--Barry Finkel --Barry Finkel
On Fri, Aug 28, 2015 at 10:10:53AM -0500, Barry S. Finkel wrote:
We've been running the RHEL5 one, which is an even older Mailman version (2.1.9), with the DMARC stuff that Red Hat merged in, and has been working fine for us (on a machine with thousands of lists, some of which are fairly active).
It's not usual to merge in this kind of added feature, but I think in this case, there was enough customer demand, and it was causing enough problems for enough people, that they made the decision to devote the resources to implement / test.
w
Will Yardley writes:
On Fri, Aug 28, 2015 at 01:37:18PM +0900, Stephen J. Turnbull wrote:
I understand that, and I agree with the logic *when you're talking about feature upgrades*. I'm just saying I don't think it's a good idea when you're dealing with a very stable package and an extreme change in the environment that effectively hamstrings the package -- they should bite the bullet and devote the resources to upgrading that package, or alternatively declare it unsupported.
We use the vendor provided Mailman package, and for the most part, it works for us, though in cases like this, it is problematic.
Precisely my point. IIUC, by the time they backported, Mailman had iterated on DMARC three times (first with an unconditional From-munging patch, then with the original patch to conditionally munge, and finally with the improved patch to deal with the preserving the ability to automatically format the response to the author). They probably also have other backported bugfixes. OTOH, Mailman 2 has basically been feature-frozen throughout the period.
Mark writes:
Stock Mailman 2.1.12 doesn't do any DMARC detection. This is quite bizarre that they would backport such a feature rather than update to 2.1.18-1 or later. Mailman 2.1 is hardly an unstable package.
You'll have to talk to the vendor who did the modifications.
What do you mean by "doesn't work"? That it doesn't go to the list at all, or that although it goes to the list, it *also* goes (separately) to the original poster?
Not going to the list at all means the MUA is broken IMO; multiple addresses in Reply-To are explicitly allowed by the RFC, and MUAs should reply to all of them.
After disabling the "add(o_from)" operation, if reply_goes_to_list is set, then (1) the reply goes *only* to the list by default, and (2) it is impossible for the user's MUA to automatically format a reply to poster. If reply_goes_to_list is not set, (1) reply by default will go to list (since it's the address in From), and (2) it is impossible to automatically format a reply to poster.
It is our opinion that (2) is a sufficiently important bug that we are unlikely to change this, even to provide an option. But if you want to make that edit, it probably does what you want. Depending on your users, you may or may not get complaints about inability to respond privately.
On 28/08/15 4:37 pm, Stephen J. Turnbull wrote:
I am not sure why the changes were back-ported - maybe it had something to do with the version of Python on the platform. I fully understand that the ability of the developers is limited to supporting known releases, and this release is not a supported one. So I don't expect more than some general guidance :)
I will try to explain, but its only on subsequent replies the issue appears:-
Step 1: email is sent to the list - the mail to DMARC protected domain is received ok by Yahoo web client. Reply-to is set to the list.
Message source of first email is:
From: Mark <mark@incet.com> Reply-To: Test <test@testdomain.org>
Step 2: Reply from Yahoo client sends an email to the list and this has the following source when viewed on Thunderbird MUA
From: Mark H via Test <test@testdomain.org> Reply-To: Mark H <markhnzy@yahoo.com>, Test <test@testdomain.org>
Step 3: A reply from the TB MUA goes to markhnzy@yahoo.com and test@testdomain.org
Because mailman helpfully recognises that a copy has already been sent to the user directly, a copy is not sent to markhnzy@ from the list.
Step 4: Now, when markhnzy@ replies, the conversation is between the recipient and the sender alone.
I agree that this is not a bug as such; its more about the desired behaviour for a specific need of mailman and that new non-RFC compliant behaviour will result.
I tried commenting out if o_from and add(o_from), but original from address was still added to the reply-to(as well as the list address). I did change some permissions and restart mailman and confirm that a new py file was created, so I don't think its that my change wasn't included.
Is there any other code in the standard mailman where the reply-to is populated with the original sender? I realise fully aware that the version of mailman I have here isn't supported, or released, and other bugs may have been introduced.
thanks again for the replies.
Mark.
Mark wrote:
That is interesting, and I wonder if this is something to do with the way that Redhat have implemented the new feature.
On my stock Mailman installation, the headers would look like this:
|From: Mark H via Test <test@testdomain.org> |Reply-To: Test <test@testdomain.org> |CC: Mark H <markhnzy@yahoo.com>
This is a much better imho way of dealing with the issue and keeps the messages flowing through the list. I know if I had the same headers on my list that you are getting, that users would be falling into the same trap, and was something I tested hard before implementing it on my lists.
Is there any way you could just upgrade your installation to the latest 2.1.20? There are some compiling guides out there for Redhat/Centos systems that were designed to get people through this before Redhat backported the feature.
Thanks. Andrew.
Sorry, serious finger fumble there. Please ignore the messed up one.
Andrew Hodgson writes:
That is interesting, and I wonder if this is something to do with the way that Redhat have implemented the new feature.
My guess is that Red Hat backported 2.1.17 or 2.1.18-1. After getting complaints, Mark S fixed things so that the From address was moved to Cc rather than Reply-To in 2.1.19 or 2.1.20.
It's easy enough to do, but beware of installing into the same places that Red Hat/CentOS uses, as that is quite likely to result in various problems if you try to upgrade the Mailman package from the OS vendor. On the other hand, using a source build installed to the default /var/local hierarchy means you'll need to copy the site config mm_cfg.py, the list configs, and the archives to the new location, and teach the webserver and mailserver about the new Mailman. In principle it's pretty straightforward and I haven't hesitated to do so on several occasions, but I can understand why an admin might be nervous (or they might have a rule that all installed software must be controlled by the system's PMS, and that's a good rule that shouldn't be violated lightly).
Mark writes:
In standard Mailman >= 2.1.18-1 (more or less) there are two places. I guess the one you found is the unconditional "from_is_list" setting in CookHeaders.py; sorry, I misled you because I was under them impression that all such changes would take place in CookHeaders. In 2.1.20, there's another, more complex path which involves checking dmarc_moderation_action in SpamDetect.py, deciding which headers to munge in CookHeaders.py:change_header(), and actually doing the work in WrapMessage.py.
I don't have 2.1.18-1 to hand, and I'm not sure that is the version that was backported anyway, so it's hard to be sure what you'd need to change. Probably searching for "dmarc" in SpamDetect and WrapMessage, and "change_header" in CookHeaders, will allow you to figure out how the backport works.
On Fri, Aug 28, 2015 at 01:37:18PM +0900, Stephen J. Turnbull wrote:
That's the way most "stable" Linux distributions do it. I think it has a lot to do with keeping the complex web of interdependencies in check, but probably also has to do with keeping features predictable. Even with security issues, generally, they go to quite a bit of work to backport things, rather than change versions.
Not sure about 6, but the specific way they backported the changes for the DMARC patch on EL5 had to do partially with dependencies... either the fact that the normal version didn't depend on dnspython and this one did, or the version of dnspython required wasn't available in EL5 (looks like they actually ended up vendoring dnspython inside the Mailman package).
Folks who want more up to date versions tend to either jump major releases more often, use third party repos, rebuild RPMs from upstream, or just build from source. We use the vendor provided Mailman package, and for the most part, it works for us, though in cases like this, it is problematic.
w
On 8/28/2015 3:03 AM, Will Yardley wrote:
It appears to me that if someone were to back-port the DMARC changes from 2.1.18 back to 2.1.12, then there is a possibility that there will be problems with the back-port due to code changes between .12 and .18. When I looked at the Ubuntu changes, I saw many changes that were not documented, and I had no idea what they did. Part of the problem in Ubuntu (or any other packager) who wants to create a package for a newer Mailman is that all of these patches, whether created to fix bugs or add features, have to be re-fitted into the code. And that takes time. I am not sure which process is more prone to introduce errors - re-fitting patches into a newer Mailman or trying to re-fit DMARC patches into on older Mailman.
This is why I determined, when I was running a Mailman installation on Ubuntu, that I would create my own package from the SourceForge source instead of installing the Ubuntu Mailman package. It took me a while to figure out how to do it, but once I did it, I had the instructions to re-do it for subsequent SF Mailman releases.
I wanted to know exactly what source I was running, so I could get assistance from the authors via this list.
--Barry Finkel --Barry Finkel
On Fri, Aug 28, 2015 at 10:10:53AM -0500, Barry S. Finkel wrote:
We've been running the RHEL5 one, which is an even older Mailman version (2.1.9), with the DMARC stuff that Red Hat merged in, and has been working fine for us (on a machine with thousands of lists, some of which are fairly active).
It's not usual to merge in this kind of added feature, but I think in this case, there was enough customer demand, and it was causing enough problems for enough people, that they made the decision to devote the resources to implement / test.
w
Will Yardley writes:
On Fri, Aug 28, 2015 at 01:37:18PM +0900, Stephen J. Turnbull wrote:
I understand that, and I agree with the logic *when you're talking about feature upgrades*. I'm just saying I don't think it's a good idea when you're dealing with a very stable package and an extreme change in the environment that effectively hamstrings the package -- they should bite the bullet and devote the resources to upgrading that package, or alternatively declare it unsupported.
We use the vendor provided Mailman package, and for the most part, it works for us, though in cases like this, it is problematic.
Precisely my point. IIUC, by the time they backported, Mailman had iterated on DMARC three times (first with an unconditional From-munging patch, then with the original patch to conditionally munge, and finally with the improved patch to deal with the preserving the ability to automatically format the response to the author). They probably also have other backported bugfixes. OTOH, Mailman 2 has basically been feature-frozen throughout the period.
participants (5)
-
Andrew Hodgson -
Barry S. Finkel -
Mark -
Stephen J. Turnbull -
Will Yardley