Mailman 2.1.18-1 not respecting add_virtualhost
Hello, we recently updated our (vhost patched) Mailman installation from 2.1.14 to 2.1.18-1 (https://launchpad.net/~msapiro) in order to workaround Yahoo's recent change to their SPF policy that this version addresses. Virtual mailing list hosting worked fine up until this upgrade.
Here is an example of our mm_cfg.py :
VIRTUAL_HOSTS.clear() DEFAULT_EMAIL_HOST = 'list.ourdomain.com' DEFAULT_URL_HOST = 'list.ourdomain.com' DEFAULT_URL_PATTERN = 'http://%s/mailman/' VIRTUAL_HOST_OVERVIEW = On add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('list.customerdomain.com', 'customerdomain.com')
If we change the default values above and run the below command, the available mailing lists move from the old default to the new.
bin/withlist -l -a -r fix_url --
All add_virtualhost configuration lines are ignored.
Since withlist is able to read the changes in our mm_cfg.py, this does not strike me as an issue with our configure options which were limited :
./configure --prefix=/usr/lib/mailman --with-var-prefix=/var/lib/mailman --with-cgi-gid=apache
I am unaware of a method we can use to further debug this, such as a command that would effectively dump out Mailman's configuration options, thus validating if it is even reading the add_virtualhost configuration lines.
--
Regards, Chris
Chris Miller President - Rocket Scientist ScratchSpace Inc. (831) 621-7928 http://www.scratchspace.com
On 08/13/2014 03:50 PM, Chris Miller wrote:
Hello, we recently updated our (vhost patched) Mailman installation from 2.1.14 to 2.1.18-1 (https://launchpad.net/~msapiro) in order to workaround Yahoo's recent change to their SPF policy that this version addresses. Virtual mailing list hosting worked fine up until this upgrade.
Here is an example of our mm_cfg.py :
VIRTUAL_HOSTS.clear() DEFAULT_EMAIL_HOST = 'list.ourdomain.com' DEFAULT_URL_HOST = 'list.ourdomain.com' DEFAULT_URL_PATTERN = 'http://%s/mailman/' VIRTUAL_HOST_OVERVIEW = On add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('list.customerdomain.com', 'customerdomain.com')
If we change the default values above and run the below command, the available mailing lists move from the old default to the new.
bin/withlist -l -a -r fix_url --
All add_virtualhost configuration lines are ignored.
Because the above command processes all lists and sets them all to the hosts corresponding to DEFAULT_URL_HOST.
To run fix_url in a multiple virtual hosts environment, you must run it separately for each list with the appropriate -u option, e.g.,
bin/withlist -l -r fix_url customer_listname -u list.customerdomain.com
I am unaware of a method we can use to further debug this, such as a command that would effectively dump out Mailman's configuration options, thus validating if it is even reading the add_virtualhost configuration lines.
bin/dumpdb lists/hostname/listname/config.pck
and look for web_page_url and host_name.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/14/2014 11:37 AM, Mark Sapiro wrote:
On 08/13/2014 03:50 PM, Chris Miller wrote:
Here is an example of our mm_cfg.py :
VIRTUAL_HOSTS.clear() DEFAULT_EMAIL_HOST = 'list.ourdomain.com' DEFAULT_URL_HOST = 'list.ourdomain.com' DEFAULT_URL_PATTERN = 'http://%s/mailman/' VIRTUAL_HOST_OVERVIEW = On add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('list.customerdomain.com', 'customerdomain.com')
If we change the default values above and run the below command, the available mailing lists move from the old default to the new.
bin/withlist -l -a -r fix_url --
All add_virtualhost configuration lines are ignored.
Because the above command processes all lists and sets them all to the hosts corresponding to DEFAULT_URL_HOST.
To run fix_url in a multiple virtual hosts environment, you must run it separately for each list with the appropriate -u option, e.g.,
bin/withlist -l -r fix_url customer_listname -u list.customerdomain.com
We did do this after I sent the original email. Unfortunately it is not changing the web_page_url in the configuration.
I am unaware of a method we can use to further debug this, such as a command that would effectively dump out Mailman's configuration options, thus validating if it is even reading the add_virtualhost configuration lines.
bin/dumpdb lists/hostname/listname/config.pck
and look for web_page_url and host_name.
The host name is correct, but the web_page_url is wrong.
Also the path we have is different than above, specifically missing a hostname directory i.e. :
lists/listname/config.pck
--
Regards, Chris
Chris Miller President - Rocket Scientist ScratchSpace Inc. (831) 621-7928 http://www.scratchspace.com
On 08/14/2014 12:07 PM, Chris Miller wrote:
On 8/14/2014 11:37 AM, Mark Sapiro wrote:
To run fix_url in a multiple virtual hosts environment, you must run it separately for each list with the appropriate -u option, e.g.,
bin/withlist -l -r fix_url customer_listname -u list.customerdomain.com
We did do this after I sent the original email. Unfortunately it is not changing the web_page_url in the configuration.
What is the output from the above command?
bin/dumpdb lists/hostname/listname/config.pck
and look for web_page_url and host_name.
The host name is correct, but the web_page_url is wrong.
Also the path we have is different than above, specifically missing a hostname directory i.e. :
lists/listname/config.pck
The vhost branch at <https://code.launchpad.net/~msapiro/mailman/vhost> creates virtual host lists with a structure of lists/hostname/listname. How did you create the list(s)?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/14/2014 12:47 PM, Mark Sapiro wrote:
On 08/14/2014 12:07 PM, Chris Miller wrote:
On 8/14/2014 11:37 AM, Mark Sapiro wrote:
To run fix_url in a multiple virtual hosts environment, you must run it separately for each list with the appropriate -u option, e.g.,
bin/withlist -l -r fix_url customer_listname -u list.customerdomain.com
We did do this after I sent the original email. Unfortunately it is not changing the web_page_url in the configuration.
What is the output from the above command?
'host_name': 'lists.customer.com',
'web_page_url': 'http://lists.ourcompany.com/mailman/',bin/dumpdb lists/hostname/listname/config.pck
and look for web_page_url and host_name.
The host name is correct, but the web_page_url is wrong.
Also the path we have is different than above, specifically missing a hostname directory i.e. :
lists/listname/config.pck
The vhost branch at <https://code.launchpad.net/~msapiro/mailman/vhost> creates virtual host lists with a structure of lists/hostname/listname. How did you create the list(s)?
We originally used the vhost patched 2.1.14 from the same repository that you maintain. I believe in that version the command line with_url option was broken, but setting these up in the Mailman GUI did the right thing in terms of setting the hostnames.
All lists were created prior to this upgrade. Looking at the tar archive of the previous installation, the /var/lib/mailman/lists directory did not contain one directory per list. Either things got moved around by "make install" or for reasons unclear to me those directories did not get backed up.
Is this simply a matter of creating the hostname directories, moving the lists into the right hostname directories, and re-running fix_url?
--
Regards, Chris
Chris Miller President - Rocket Scientist ScratchSpace Inc. (831) 621-7928 http://www.scratchspace.com
On 08/14/2014 01:17 PM, Chris Miller wrote:
What is the output from the above command?
'host_name': 'lists.customer.com', 'web_page_url': 'http://lists.ourcompany.com/mailman/',
I meant the actual command output, not the result. I.e., something like
Importing fix_url... Running fix_url.fix_url()... Loading list listname (locked) Saving list Finalizing
but perhaps with some traceback or other info.
Is this simply a matter of creating the hostname directories, moving the lists into the right hostname directories, and re-running fix_url?
The entire purpose of the vhost branch is to add host_name subdirectories to the lists/ directory so that list names do not have to be globally unique. If this in not a requirement in your case, I suggest installing the unpatched 2.1.18-1 release.
The vhost has issues. In particular bin/withlist -l does not work with hostname/listname type lists.
If all your listname directories are directly under lists/ and your mm_cfg.py contains
add_virtualhost('list.customerdomain.com', 'customerdomain.com')
I don't know why
bin/withlist -l -r fix_url listname -u list.customerdomain.com
would fail to set the list's web_page_url to http://list.customerdomain.com/mailman/.com/mailman/ unless there is some typo in the domain name in the withlist command or the add_virtualhost line.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/14/2014 5:31 PM, Mark Sapiro wrote:
On 08/14/2014 01:17 PM, Chris Miller wrote:
What is the output from the above command?
'host_name': 'lists.customer.com', 'web_page_url': 'http://lists.ourcompany.com/mailman/',I meant the actual command output, not the result. I.e., something like
Importing fix_url... Running fix_url.fix_url()... Loading list listname (locked) Saving list Finalizing
but perhaps with some traceback or other info.
I sent you a PM with output.
Is this simply a matter of creating the hostname directories, moving the lists into the right hostname directories, and re-running fix_url?
The entire purpose of the vhost branch is to add host_name subdirectories to the lists/ directory so that list names do not have to be globally unique. If this in not a requirement in your case, I suggest installing the unpatched 2.1.18-1 release.
We do want unique names, but in practice this never really worked right. All of the lists do need to maintain their own unique domain name space including access to the list, so we need to sort this out.
We started with the patches from the original contributor and version 2.1.5. I think we went to 2.1.9, 2.1.14, and now the latest. It's possible that along the way issues were introduced, but this has more or less worked properly.
The vhost has issues. In particular bin/withlist -l does not work with hostname/listname type lists.
If all your listname directories are directly under lists/ and your mm_cfg.py contains
add_virtualhost('list.customerdomain.com', 'customerdomain.com')
I don't know why
I am confused as well. To restate the question is it possible I just need to create the hostname directories and reorganize everything?
--
Regards, Chris
Chris Miller President - Rocket Scientist ScratchSpace Inc. (831) 621-7928 http://www.scratchspace.com
On 08/14/2014 06:06 PM, Chris Miller wrote:
On 8/14/2014 5:31 PM, Mark Sapiro wrote:
I sent you a PM with output.
OK. I'll respond when it gets through my greylisting.
The vhost has issues. In particular bin/withlist -l does not work with hostname/listname type lists.
Actually, it does. It's been so long since I looked at this, I forgot. the command for a list whose config directory is hostname/listname is
bin/withlist -l -r fix_url listname@hostname -u web_hostname
I am confused as well. To restate the question is it possible I just need to create the hostname directories and reorganize everything?
That might work, but fix_url should work on a list whose directory is directly under lists/. The mystery here is it apparently doesn't.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 8/14/2014 6:21 PM, Mark Sapiro wrote:
On 08/14/2014 06:06 PM, Chris Miller wrote:
On 8/14/2014 5:31 PM, Mark Sapiro wrote:
I sent you a PM with output.
OK. I'll respond when it gets through my greylisting.
The vhost has issues. In particular bin/withlist -l does not work with hostname/listname type lists.
Actually, it does. It's been so long since I looked at this, I forgot. the command for a list whose config directory is hostname/listname is
bin/withlist -l -r fix_url listname@hostname -u web_hostname
I am confused as well. To restate the question is it possible I just need to create the hostname directories and reorganize everything?
That might work, but fix_url should work on a list whose directory is directly under lists/. The mystery here is it apparently doesn't.
Sorry to re-ignite this so late. To recap, the problem with our Mailman config is that all mailing lists were created under /var/lib/mailman/lists instead of /var/lib/mailman/lists/<hostname>.
We have corrected this problem by creating the subdirectories, moving the lists in to the proper sub directory, and setting permissions. We are still having trouble with fix_url.
If we run this the traditional way, we get an error :
root@mail /var/lib/mailman/lists $ ~mailman/bin/withlist -l -r fix_url brewclubtest -u lists.fakedomain.com Importing fix_url... Running fix_url.fix_url()... Loading list brewclubtest (locked) Unknown list: brewclubtest Traceback (most recent call last): File "/usr/lib/mailman/bin/withlist", line 299, in <module> main() File "/usr/lib/mailman/bin/withlist", line 277, in main r = do_list(listname, args, func) File "/usr/lib/mailman/bin/withlist", line 202, in do_list return func(m, *args) File "/usr/lib/mailman/bin/fix_url.py", line 73, in fix_url if not mlist.Locked(): AttributeError: 'NoneType' object has no attribute 'Locked'
If we use the <listname@hostname> method, the command completes correctly, but then we have issues with the web interface :
root@mail /var/lib/mailman/lists $ ~mailman/bin/withlist -l -r fix_url brewclubtest@lists.brewclub.org -u lists.fakedomain.com Importing fix_url... Running fix_url.fix_url()... Loading list brewclubtest@lists.fakedomain.com (locked) Saving list Finalizing
When we go to the list page here :
http://lists.fakedomain.com/mailman/admin/brewclubtest
The list is present, but there is an error in red at the top of the screen that says "No such list brewclubtest@fakedomain.com". Please note that the hostname "lists" was stripped from the error.
We are able to login to the list URL which now contains <listname@hostname> in the URL :
http://lists.fakedomain.com/mailman/admin/brewclubtest@lists.fakedomain.com
But clicking any links in the admin interface appends @fakedomain.com which results in a new login prompt :
http://lists.fakedomain.com/mailman/admin/brewclubtest@lists.fakedomain.com@...
It seems like fix_url needs to be patched to use the traditional method of using only the list URL, and/or needs to be patched to properly handle the <listname@hostname> method.
To clarify, mm_cfg.py does contain the correct info for the lists, this all broke during the upgrade to 2.1.18 to utilize the DMARC features.
VIRTUAL_HOSTS.clear() DEFAULT_URL_PATTERN = 'http://%s/mailman/' VIRTUAL_HOST_OVERVIEW = On add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('lists.fakedomain.com', 'fakedomain.com')
Thoughts?
--
Regards, Chris
Chris Miller President - Rocket Scientist ScratchSpace Inc. (831) 621-7928 http://www.scratchspace.com
On 07/01/2015 12:07 PM, Chris Miller wrote:
We have corrected this problem by creating the subdirectories, moving the lists in to the proper sub directory, and setting permissions. We are still having trouble with fix_url.
I'm just confirming that I've seen this and will look at it at some point this week, but not immediately.
It may be that fix_url never did the right thing, but I need to look in detail.
I think however that you have made a mistake in that both the subdirectory name in the lists/ directory and the domain part of the listname in the fix_url command should be the email host, not the web host. I.e. no lists. prefix, at least with settings like
add_virtualhost('lists.fakedomain.com', 'fakedomain.com')
To clarify, mm_cfg.py does contain the correct info for the lists, this all broke during the upgrade to 2.1.18 to utilize the DMARC features.
What exactly are you running? Are you running code from <https://code.launchpad.net/~msapiro/mailman/vhost>? I wouldn't be too surprised if it's broken. I think the last thing I actually did to that branch other than untested merges of the upstream branch upon releases was over two years ago.
I will look into it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 7/1/2015 6:24 PM, Mark Sapiro wrote:
On 07/01/2015 12:07 PM, Chris Miller wrote:
We have corrected this problem by creating the subdirectories, moving the lists in to the proper sub directory, and setting permissions. We are still having trouble with fix_url.
I'm just confirming that I've seen this and will look at it at some point this week, but not immediately.
Awesome. Much appreciated.
It may be that fix_url never did the right thing, but I need to look in detail.
I think however that you have made a mistake in that both the subdirectory name in the lists/ directory and the domain part of the listname in the fix_url command should be the email host, not the web host. I.e. no lists. prefix, at least with settings like
add_virtualhost('lists.fakedomain.com', 'fakedomain.com')
I renamed the directory to just "fakedomain.com" and re-ran fix_url. I no longer see the red error message in the web interface, however once logged in I continue to see @fakedomain appended in the URL when clicking any navigation item. While I can login again with the invalid URL, each navigation click continues to append @fakedomain in a cumulative fashion.
To clarify, mm_cfg.py does contain the correct info for the lists, this all broke during the upgrade to 2.1.18 to utilize the DMARC features.
What exactly are you running? Are you running code from <https://code.launchpad.net/~msapiro/mailman/vhost>?
Yes. 2.1.18-1
http://bazaar.launchpad.net/~msapiro/mailman/vhost/revision/868
I wouldn't be too surprised if it's broken. I think the last thing I actually did to that branch other than untested merges of the upstream branch upon releases was over two years ago.
We downloaded this version last year to utilize the DMARC patches after Yahoo changed their SPF policy and our mailman posts were getting rejected.
--
Regards, Chris
Chris Miller President - Rocket Scientist ScratchSpace Inc. (831) 621-7928 http://www.scratchspace.com
On 07/02/2015 03:33 PM, Chris Miller wrote:
I renamed the directory to just "fakedomain.com" and re-ran fix_url. I no longer see the red error message in the web interface, however once logged in I continue to see @fakedomain appended in the URL when clicking any navigation item. While I can login again with the invalid URL, each navigation click continues to append @fakedomain in a cumulative fashion.
I do not see this in my test/development installation.
I have however made a few other changes:
- Added '@' to ACCEPTABLE_LISTNAME_CHARACTERS because of fix for LP: #1437145.
- Changed fix_url.py to better handle vhost lists.
- Fixed CSRFcheck.py to account for '@' to '%40' changes.
- Fixed generated public archive URL for vhost lists.
The fix_url changes make it more robust in updating lists with '@' in the name.
In any case, I recommend getting the latest rev. 874 from <http://bazaar.launchpad.net/~msapiro/mailman/vhost/revision/874> and upgrading to that.
As far as fix_url is concerned, there are four attributes it is concerned with, two of which are vhost only. You can see these if you do (for listname@emaildomain in the web domain webdomain)
bin/dumpdb lists/emaildomain/listname/config.pck
you should see
hostname = 'emaildomain' list_address = 'listname@emaildomain' local_part = 'listname' web_page_url = 'http://webdomain/mailman/'
(the last assumes the default setting for DEFAULT_URL_PATTERN)
If you install rev. 874 and you still have the spurious appending of '@emaildomain' to URLs issue, I will look further. I suspect there is some issue with older code assuming the list name is just the short name and appending '@emaildomain' to 'correct' the URL, but as I say, I don't see it in my system.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Chris Miller -
Mark Sapiro