Hello,
I work for a large university and was recently put in charge of consolidating a bunch of mailman lists from multiple domains onto 1 server.
First problem that I encountered was the single namespace issue with virtual domains, so i removed the stock Mailman 2.1.12 that came with Ubuntu, downloaded Mailman 2.1.7 and patched it with mailman-2.1.7-release-to-vhost.patch from http://ndim.fedorapeople.org/stuff/mailman-vhost/.
That went pretty smooth, compiled and installed without any major issues.
put these in mm_cfg.py: DEFAULT_EMAIL_HOST = 'lists.dept.university.edu' DEFAULT_URL_HOST = 'lists.dept.university.edu' add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('lists.domain1.net', 'lists.domain1.net') add_virtualhost('lists.domain2.net', 'lists.domain2.net') POSTFIX_STYLE_VIRTUAL_DOMAINS = ['lists.domain1.net', 'lists.domain2.net']
After that I setup Postfix, added a new list "test-list" which was automatically created in the right place - /path/to/mailman/lists/lists.domain1.net/test-list/ So no problem here. I subscribed to the list, confirmed the subscription and sent couple of test messages. Everything worked great.
Next i went to browse the archives and encountered another problem. The URLs for the Public archives were not right, but I got around it with a simple Apache rewrite:
RewriteCond %{REQUEST_URI} !^/pipermail/([a-z]+)\.(.*)?$ [NC]
RewriteRule ^/pipermail/(.*)$ /pipermail/%{SERVER_NAME}/$1 [R=301,L]Basically all it does is insert "lists.domainX.net" between /pipermail/ and /test-list/. Public archives are now working with http://lists.domain1.net/pipermail/lists.domain1.net/test-list/
Next I tried to test Private archives and this is where I hit the wall and need some help with.
The listinfo page for my test-list showed that the URL for the private archives was http://lists.domain1.net/private/test-list, so i figured i would be able to fix it with the same rewrite rule that i used for the Public archives. However, because private CGI wrapper expects the list name right after /private/, this URL http://lists.domain1.net/private/lists.domain1.com/test-list/ did not work, and all i got was the error message saying that "list lists.domain1.net does not exist"
Next, I changed my rewrite rule to redirect me to http://lists.domain1.com/private/test-list@lists.domain1.com instead
RewriteCond %{REQUEST_URI} !^/private/(.*)@(.*)?$ [NC]
RewriteRule ^/private/(.*)/$ /private/$1@%{SERVER_NAME} [R=301,L]This redirected me to the login page and seemed to work great, until I tried to login and got hit with the message that "test-list" does not exist.
After a little digging, I traced the problem to the incorrect html from action, private.py and private.html template file. With my very limited [read "non-existing"] Python knowledge I managed to pass some arguments to the private.html and change the form action from http://lists.domain1.com/private/test-list/ to http://lists.domain1.com/private/test-list@lists.domain1.com and was able to login, but now i got another error - "Private archive file not found". I looked at the log file and saw the following error message:
Jan 26 09:31:18 2010 (21196) Private archive file not found: /path/to/mailman/archives/private/test-list@lists.domain1.net
Just for the hell of it I decided to create a symbolic link and see if it will work
ln -s /path/to/mailman/archives/private/lists.domain1.net/test-list /path/to/mailman/archives/private/test-list@lists.domain1.netThis seemed to work and i was able to pull up the Private archive page, however all the links were missing "/private/test-list@lists.domain1.net" and looked like "http://lists.domain1.com/2010-January/" [this part is not really important right now. i guess i will deal with this problem later.]
Now it looks like the "path" variable in private.py (line 70 or very close) is causing the issue:
path = os.environ.get('PATH_INFO')According to the debug log my PATH_INFO is test-list@lists.domain1.net, so I figured I should be able to fix it like so:
path = os.path.join(os.environ.get('SERVER_NAME'), <LIST NAME HERE>) [ end result has to be "lists.domain1.net/test-list ]But i could not figure out how to pass the list name to os.path.join() and no matter what I did i either ended up with undeclared variable or some other related error.
I really hope someone would be able to help me out here or point me in the right direction.
Thank you in advance,
- Igor
UGSD wrote:
First problem that I encountered was the single namespace issue with virtual domains, so i removed the stock Mailman 2.1.12 that came with Ubuntu, downloaded Mailman 2.1.7 and patched it with mailman-2.1.7-release-to-vhost.patch from http://ndim.fedorapeople.org/stuff/mailman-vhost/.
That went pretty smooth, compiled and installed without any major issues.
Before you commit too heavily to this, I suggest you look at the first 570+ lines at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/annotate/head%3A/NEW...> and see if you really want to be installing a 4 year old release.
put these in mm_cfg.py: DEFAULT_EMAIL_HOST = 'lists.dept.university.edu' DEFAULT_URL_HOST = 'lists.dept.university.edu' add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('lists.domain1.net', 'lists.domain1.net') add_virtualhost('lists.domain2.net', 'lists.domain2.net') POSTFIX_STYLE_VIRTUAL_DOMAINS = ['lists.domain1.net', 'lists.domain2.net']
After that I setup Postfix, added a new list "test-list" which was automatically created in the right place - /path/to/mailman/lists/lists.domain1.net/test-list/ So no problem here. I subscribed to the list, confirmed the subscription and sent couple of test messages. Everything worked great.
Next i went to browse the archives and encountered another problem. The URLs for the Public archives were not right, but I got around it with a simple Apache rewrite:
RewriteCond %{REQUEST_URI} !^/pipermail/([a-z]+)\.(.*)?$ [NC] RewriteRule ^/pipermail/(.*)$ /pipermail/%{SERVER_NAME}/$1 [R=301,L]
Basically all it does is insert "lists.domainX.net" between /pipermail/ and /test-list/. Public archives are now working with http://lists.domain1.net/pipermail/lists.domain1.net/test-list/
So you can't have lists with names beginning with letters and a dot.
I think a better method would be a separate Alias /pipermail/ directive in each VirtualHost block.
Next I tried to test Private archives and this is where I hit the wall and need some help with.
The listinfo page for my test-list showed that the URL for the private archives was http://lists.domain1.net/private/test-list, so i figured i would be able to fix it with the same rewrite rule that i used for the Public archives.
The patch is supposed to present the correct URL for both private and public archive access. I probably won't have time before Thursday to look at this in detail (and I only do it at all because OCD is a terrible thing to waste).
[long sad tale snipped]
You are applying band-aids upon band-aids and you'll never stop the bleeding until you properly fix the underlying problem.
As I said, because I'm compulsive I will look at this in more detail and follow up, but probably not for a few days.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Before you commit too heavily to this, I suggest you look at the first 570+ lines at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/annotate/head%3A/NEW...> and see if you really want to be installing a 4 year old release.
The only reason i chose 2.1.7 is because the patch applied cleanly, but i guess you are right. I just sat down and applied the patch to the 2.1.13 source. There few few rejects, but I went through all of them and it looks like that code that already made it upstream and included in 2.1.13.
Compiled and installed it, but for the life of me i cant create a new list for a virtual domain.
bin/newlist -u lists.domain1.net test-list@lists.domain1.netand end up with
Illegal list name: pruchai-test@lists.calit2.netThis is very strange, because the domain already exists in mm_cfg.py and the patch applied cleanly to bin/newlist too. Any idea what may be wrong?
put these in mm_cfg.py: DEFAULT_EMAIL_HOST = 'lists.dept.university.edu' DEFAULT_URL_HOST = 'lists.dept.university.edu' add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST) add_virtualhost('lists.domain1.net', 'lists.domain1.net') add_virtualhost('lists.domain2.net', 'lists.domain2.net') POSTFIX_STYLE_VIRTUAL_DOMAINS = ['lists.domain1.net', 'lists.domain2.net']
RewriteCond %{REQUEST_URI} !^/pipermail/([a-z]+)\.(.*)?$ [NC] RewriteRule ^/pipermail/(.*)$ /pipermail/%{SERVER_NAME}/$1 [R=301,L]
Basically all it does is insert "lists.domainX.net" between /pipermail/ and /test-list/. Public archives are now working with http://lists.domain1.net/pipermail/lists.domain1.net/test-list/
So you can't have lists with names beginning with letters and a dot.
I think a better method would be a separate Alias /pipermail/ directive in each VirtualHost block.
Right. That was a quick rule and i already realized that couple of my lists are not gonna work with this. There is, however, a common name convention, so i will make a better match later.
As far as virtual hosts go, I'd like to keep configuration to a minimum. As i mentioned on IRC, I am looking at approximately 20 separate domains and maintaining 20 separate installations and configs is a management nightmare.
Next I tried to test Private archives and this is where I hit the wall and need some help with.
The listinfo page for my test-list showed that the URL for the private archives was http://lists.domain1.net/private/test-list, so i figured i would be able to fix it with the same rewrite rule that i used for the Public archives.
As I said, because I'm compulsive I will look at this in more detail and follow up, but probably not for a few days.
Thanks, Mark! I really appreciate the help.
UGSD quoted me and wrote:
Before you commit too heavily to this, I suggest you look at the first 570+ lines at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/annotate/head%3A/NEW...> and see if you really want to be installing a 4 year old release.
The only reason i chose 2.1.7 is because the patch applied cleanly, but i guess you are right. I just sat down and applied the patch to the 2.1.13 source. There few few rejects, but I went through all of them and it looks like that code that already made it upstream and included in 2.1.13.
I made a 2.1.13 vhost patch a little differently. I made a bazaar 2.1.7 branch, applied the 2.1.7 vhosts patch, committed the result, did a bzr merge from the 2.1.13 branch which produced 5 conflicts, the only significant ones being in Mailman/MTA/Utils.py and Mailman/Utils.py, and resolved the conflicts.
I am testing with that. This patch is at <http://www.msapiro.net/mm/2.1.13_vhost.patch> if you're interested. It is essentially the 2.1.7 patch, but fixed to apply cleanly to 2.1.13.
Compiled and installed it, but for the life of me i cant create a new list for a virtual domain.
bin/newlist -u lists.domain1.net test-list@lists.domain1.netand end up with
Illegal list name: pruchai-test@lists.calit2.netThis is very strange, because the domain already exists in mm_cfg.py and the patch applied cleanly to bin/newlist too. Any idea what may be wrong?
Yes. Really the MailList.Create() method should be patched, but the workaround is to add
ACCEPTABLE_LISTNAME_CHARACTERS = '[-+_.=a-z0-9@]'
to mm_cfg.py. This adds '@' to the default list of acceptable listname characters and allows list names with '@' to pass a new (since 2.1.7) check in MailList.Create().
As far as the problems with the archives are concerned, I've tested just a bit and I see the problem (but not the solution yet). I think part of the problem for me is that the host name in a listname@hostname list cannot be the same as DEFAULT_URL_HOST. I have a DNS issue on my test machine that makes it difficult for me to add additional host names so that's hampering my testing at the moment.
But the issue is more than that. At the moment, I have four lists - two lists.listname lists and two lists/msapiro.net/listname lists. All four of these appear on the overviews at http://msapiro.net/mailman/admin and http://msapiro.net/mailman/listinfo with link URLs like http://msapiro.net/mailman/admin/listname and http://msapiro.net/mailman/listinfo/listname, but only the lists/msapiro.net/listname list URLs work. The others give "No such list listname@msapiro.net".
I will continue to look at this, but if your testing turns up anything, let us know.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark,
I made a 2.1.13 vhost patch a little differently. I made a bazaar 2.1.7 branch, applied the 2.1.7 vhosts patch, committed the result, did a bzr merge from the 2.1.13 branch which produced 5 conflicts, the only significant ones being in Mailman/MTA/Utils.py and Mailman/Utils.py, and resolved the conflicts.
I am testing with that. This patch is at <http://www.msapiro.net/mm/2.1.13_vhost.patch> if you're interested. It is essentially the 2.1.7 patch, but fixed to apply cleanly to 2.1.13.
Thank you for taking a look at this. I really appreciate it.
Yes. Really the MailList.Create() method should be patched, but the workaround is to add
ACCEPTABLE_LISTNAME_CHARACTERS = '[-+_.=a-z0-9@]'
to mm_cfg.py. This adds '@' to the default list of acceptable listname characters and allows list names with '@' to pass a new (since 2.1.7) check in MailList.Create().
Yep, i figured this one out.
As far as the problems with the archives are concerned, I've tested just a bit and I see the problem (but not the solution yet). I think part of the problem for me is that the host name in a listname@hostname list cannot be the same as DEFAULT_URL_HOST. I have a DNS issue on my test machine that makes it difficult for me to add additional host names so that's hampering my testing at the moment.
I got around the DNS issue with /etc/hosts, may not work in your case, i guess
After not being able to create a list, I had another issue (not seen in 2.1.7), where the alias file was created in the form of
lists.domain1.net=list-name-unsubscribe: "|/path/to/mailman unsubscribe lists.domain1.net=list-name"
which mailman did not understand and complained that "lists.domain1.net=list-name" not found.
The problem was in Mailman/MTA/Utils.py and _makealiases_mailprog() function, so i just modified it to accept another variable and use it
def _makealiases_mailprog(listname, at_name, internal_listname=None): <<--- i added at_name
for ext in ('admin', 'bounces', 'confirm', 'join', 'leave', 'owner',
'request', 'subscribe', 'unsubscribe'):
aliases.append(('%s-%s' % (listname, ext),
'"|%s %s %s"' % (wrapper, ext, at_name))) <<--- i added at_nameand then change Mailman/MTA/Postfix.py
for k, v in makealiases(_generate_alias(mlist), listname, <<--- added listname var here
internal_listname = listname):This fixed the alias creation. I was able to subscribe to the list, however, i was not able to post to the list, because list names were still translated into "lists.domain1.net=list-name". Tacked it down to the post script in scripts folder, and had to modify "listname" variable in mail()
try:
foo = sys.argv[1]
listname = foo.split('=')
listname.reverse()
listname = '@'.join(listname)This change allowed me to post, but this is where i hit the same problems as i did originally with 2.1.7 install.
So i had to use my mod_rewrite trick, modify Mailman/Cgi/pryvate.py
host = os.environ.get('SERVER_NAME') <<----- needed to for Utils.maketext()
and
list_name = listname.split("@")
m_list = list_name[0]
true_filename = os.path.join(
mm_cfg.PRIVATE_ARCHIVE_FILE_DIR, host, m_list)and then modify templates/en/private.html and change form action, it it allows me to authenticate private archives.
<FORM METHOD=POST ACTION="%(action)s@%(host)s">
But the issue is more than that. At the moment, I have four lists - two lists.listname lists and two lists/msapiro.net/listname lists. All four of these appear on the overviews at http://msapiro.net/mailman/admin and http://msapiro.net/mailman/listinfo with link URLs like http://msapiro.net/mailman/admin/listname and http://msapiro.net/mailman/listinfo/listname, but only the lists/msapiro.net/listname list URLs work. The others give "No such list listname@msapiro.net".
Now i was able to see the private archive table of contents, but none of the links were functional, so clicking on a link pointing to
http://lists.domain1.net/private/list-name@lists.domain1.net/2010-January/th... reloads the page, adding another "2010-Janury" in the browser address bar
http://lists.domain1.net/private/list-name@lists.domain1.net/2010-January/20... http://lists.domain1.net/private/list-name@lists.domain1.net/2010-January/20...
and so on. Unfortunately i haven't been able to get around this issue yet.
I will continue to look at this, but if your testing turns up anything, let us know.
Thanks again, Mark!
-Igor
UGSD wrote:
I got around the DNS issue with /etc/hosts, may not work in your case, i guess
Yeah, I can do that too. I've actually created another domain that way, but I havent gotten back to debugging yet.
After not being able to create a list, I had another issue (not seen in 2.1.7), where the alias file was created in the form of
lists.domain1.net=list-name-unsubscribe: "|/path/to/mailman unsubscribe lists.domain1.net=list-name"
which mailman did not understand and complained that "lists.domain1.net=list-name" not found.
The problem was in Mailman/MTA/Utils.py and _makealiases_mailprog() function, so i just modified it to accept another variable and use it
I'm not sure where the '=' comes from, and I don't know if this is the issue, but Mailman/MTA/Utils.py is one of two modules that had a real conflict when I merged the patches with 2.1.13. The attached MTA_Utils.py.patch.txt patch is (I think) the proper fix. The secton at "@@ -48,13 +54,12 @@" is where the problem was. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
UGSD wrote:
After not being able to create a list, I had another issue (not seen in 2.1.7), where the alias file was created in the form of
lists.domain1.net=list-name-unsubscribe: "|/path/to/mailman unsubscribe lists.domain1.net=list-name"
which mailman did not understand and complained that "lists.domain1.net=list-name" not found.
Alias creation works just fine for me with my original 2.1.13 patch at <http://www.msapiro.net/mm/2.1.13_vhost.patch>. Lists in the DEFAULT_URL_HOST domain get aliases like:
# STANZA START: mailman # CREATED: Sun Jan 31 12:57:36 2010 mailman: "|/usr/local/mailman/mail/mailman post mailman" mailman-admin: "|/usr/local/mailman/mail/mailman admin mailman" mailman-bounces: "|/usr/local/mailman/mail/mailman bounces mailman" etc.
Lists in the virtual domains get aliases like
# STANZA START: test@lists.msapiro.net # CREATED: Sun Jan 31 12:57:37 2010 lists.msapiro.net=test: "|/usr/local/mailman/mail/mailman post test@lists.msapiro.net" lists.msapiro.net=test-admin: "|/usr/local/mailman/mail/mailman admin test@lists.msapiro.net" lists.msapiro.net=test-bounces: "|/usr/local/mailman/mail/mailman bounces test@lists.msapiro.net" etc.
and entries in virtual-mailman like
# STANZA START: test@lists.msapiro.net # CREATED: Sun Jan 31 12:57:37 2010 test@lists.msapiro.net lists.msapiro.net=test test-admin@lists.msapiro.net lists.msapiro.net=test-admin test-bounces@lists.msapiro.net lists.msapiro.net=test-bounces
This is as it should be. The only requirement being that the virtual domains (lists.msapiro.net in this example) be in the POSTFIX_STYLE_VIRTUAL_DOMAINS list in mm_cfg.py as in:
POSTFIX_STYLE_VIRTUAL_DOMAINS = ['lists.msapiro.net']
So i had to use my mod_rewrite trick, modify Mailman/Cgi/pryvate.py
[...]
Now i was able to see the private archive table of contents, but none of the links were functional, so clicking on a link pointing to
http://lists.domain1.net/private/list-name@lists.domain1.net/2010-January/th... reloads the page, adding another "2010-Janury" in the browser address bar
http://lists.domain1.net/private/list-name@lists.domain1.net/2010-January/20... http://lists.domain1.net/private/list-name@lists.domain1.net/2010-January/20...
and so on. Unfortunately i haven't been able to get around this issue yet.
I fixed the archiving for both private and public archives so no rewriting is involved. private.py does the right thing with my additional patches. I also patched the GetBaseArchiveURL() method to include the virtual host directory in the public archive URL.
Finally, I had an issue which almost caused me to give up in disgust. The original patch adds a _GetURLHost() function to Mailman.Utils.py which gets the host name from the invoking URL. This got the name from the SERVER_NAME environment variable which is not the preferred place and is different from the URL host in my test environment. It should use HTTP_HOST.
The bottom line for me now is I can take a 2.1.13 base distribution, patch it with the patch at <http://www.msapiro.net/mm/2.1.13_vhost.patch> and patch that with the vhost_extra_patch.txt patch attached to this post, and I think that list creation and the web interface work. I haven't tested posting because my testbed uses Exim, but I think it should work.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark,
I fixed the archiving for both private and public archives so no rewriting is involved. private.py does the right thing with my additional patches. I also patched the GetBaseArchiveURL() method to include the virtual host directory in the public archive URL.
Finally, I had an issue which almost caused me to give up in disgust. The original patch adds a _GetURLHost() function to Mailman.Utils.py which gets the host name from the invoking URL. This got the name from the SERVER_NAME environment variable which is not the preferred place and is different from the URL host in my test environment. It should use HTTP_HOST.
The bottom line for me now is I can take a 2.1.13 base distribution, patch it with the patch at <http://www.msapiro.net/mm/2.1.13_vhost.patch> and patch that with the vhost_extra_patch.txt patch attached to this post, and I think that list creation and the web interface work. I haven't tested posting because my testbed uses Exim, but I think it should work.
Thank you sooo much! Both of these patches worked perfectly and now i have a working, fully virtual mailman installation. This is not something i would've been able to figure out of my own. The only thing it complained about is --with-mail-gid. My previous installation attempts worked fine with --with-mail-gid=list, but this time it complained that it needs --with-mail-gid=nobody, but that was an easy fix.
Thanks again, Mark. I really appreciate your help and I am sure that many other users would benefit greatly from these 2 patches.
- Igor
participants (2)
-
Mark Sapiro -
UGSD