Some detail questions about migrating lists
Hi all,
I need to migrate lists from an *old* Slackware system running MM 2.0 (0x200000f0) to a new system where MM is installed via RPM (2.1.9).
I've read the posts linked to by the FAQ, and the basic procedure is to copy the data, lists, and archives directories and run withlist fix_url. But I have a couple of minor questions:
According to the UPGRADE document, file formats have changed as have actual files (some new, some gone). What triggers the 2.1 version to do the necessary conversions ? Is it just finding files of the old version and recognizing them as such ? Does the last_mailman_version in the data directory trigger this ?
There are various minor concerns, such as file permissions, aliases, redoing public archive symlinks, and so forth. This made me wonder if it would be easier/possible to run bin/newlist for each existing 2.0 list in the 2.1.9 system, and *then* bring over the data, lists, and archives/private directories. Advisable ?
TIA
Frank Griffin wrote:
According to the UPGRADE document, file formats have changed as have actual files (some new, some gone). What triggers the 2.1 version to do the necessary conversions ? Is it just finding files of the old version and recognizing them as such ?
Yes, as far as the old config.db to new config.pck conversion is concerned.
As far as other files such as the pending database and qfiles are concerned, these will be converted by bin/update if you are upgrading an installation, but if you are simply moving lists from an old to a new installation, it is best to deal with pending requests and queued messages on the old system and blow off anything that is left. There is no automated way of converting these in the new installation.
Does the last_mailman_version in the data directory trigger this ?
No. last_mailman_version tells bin/update whether it needs to do anything. This comes into play when installing an upgrade, but when moving an old list's config.(db|pck) into an existing installation, it is the DATA_FILE_VERSION in the config file itself that triggers the update.
There are various minor concerns, such as file permissions, aliases, redoing public archive symlinks, and so forth. This made me wonder if it would be easier/possible to run bin/newlist for each existing 2.0 list in the 2.1.9 system, and *then* bring over the data, lists, and archives/private directories. Advisable ?
No. This is absolutely wrong. If you create the list on the new system first, Mailman will always see that list's config.pck and will never look at the config.db that you move over.
My advice is:
Do not create the lists on the new host.
Do not move anything from the data/ directory.
Move only the config.db from the lists/<listname>/ directories.
Move the archives/private/ tree.
Run bin/list_lists which will instantiate every list and convert every old config.db to a new config.pck.
Run bin/genaliases to create/update aliases.
After conversion, remove the config.db files as if you don't, a possible future problem in reading the config.pck and config.pck.last files could cause Mailman to fall back to the then outdated config.db.
Public archive symlinks will be created automatically the first time a list's configuration is saved. The bin/list_lists accessing and converting of the config.db files won't do this, but a list post or accessing the admin web interface for a list will.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
Frank Griffin wrote:
According to the UPGRADE document, file formats have changed as have actual files (some new, some gone). What triggers the 2.1 version to do the necessary conversions ? Is it just finding files of the old version and recognizing them as such ?
Yes, as far as the old config.db to new config.pck conversion is concerned.
As far as other files such as the pending database and qfiles are concerned, these will be converted by bin/update if you are upgrading an installation, but if you are simply moving lists from an old to a new installation, it is best to deal with pending requests and queued messages on the old system and blow off anything that is left. There is no automated way of converting these in the new installation.
Does the last_mailman_version in the data directory trigger this ?
No. last_mailman_version tells bin/update whether it needs to do anything. This comes into play when installing an upgrade, but when moving an old list's config.(db|pck) into an existing installation, it is the DATA_FILE_VERSION in the config file itself that triggers the update.
There are various minor concerns, such as file permissions, aliases, redoing public archive symlinks, and so forth. This made me wonder if it would be easier/possible to run bin/newlist for each existing 2.0 list in the 2.1.9 system, and *then* bring over the data, lists, and archives/private directories. Advisable ?
No. This is absolutely wrong. If you create the list on the new system first, Mailman will always see that list's config.pck and will never look at the config.db that you move over.
My advice is:
Do not create the lists on the new host.
Do not move anything from the data/ directory.
Move only the config.db from the lists/<listname>/ directories.
Move the archives/private/ tree.
Run bin/list_lists which will instantiate every list and convert every old config.db to a new config.pck.
Run bin/genaliases to create/update aliases.
After conversion, remove the config.db files as if you don't, a possible future problem in reading the config.pck and config.pck.last files could cause Mailman to fall back to the then outdated config.db.
Public archive symlinks will be created automatically the first time a list's configuration is saved. The bin/list_lists accessing and converting of the config.db files won't do this, but a list post or accessing the admin web interface for a list will.
Mark,
Thanks for the reply. I've recently gotten back to this and wanted to post my results.
In addition to the steps you posted, I needed to run "withlist -a -l -r fix_url".
However, I have one anomaly that still needs resolution:
All of the lists I moved had the option set to reply to the list (yes, I know), and they show up this way when I display them in the admin UI. However, when I post to them, the post mails arrive with a reply-to of the poster and the list description linked to an email address of the list owner, e.g.
List-Id: WebData Project Discussion <webdata.ftgme2.griffin.selectbs.com> List-Unsubscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/webdata>, <mailto:webdata-request@ftgme2.griffin.selectbs.com?subject=unsubscribe> List-Archive: <http://ftgme2.griffin.selectbs.com/pipermail/webdata> List-Post: <mailto:webdata@ftgme2.griffin.selectbs.com> List-Help: <mailto:webdata-request@ftgme2.griffin.selectbs.com?subject=help> List-Subscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/webdata>, <mailto:webdata-request@ftgme2.griffin.selectbs.com?subject=subscribe> Reply-To: ftg@roadrunner.com, WebData Project Discussion <frank.griffin@selectbs.com>
In this case, the list owner is frank.griffin@selectbs.com and the poster is ftg@roadrunner.com . It seems to me that this ought to be WebData Project Discussion <webdata@ftgme2.griffin.selectbs.com> .
Did I miss a step ? What can I do to correct this ?
Frank Griffin wrote:
In this case, the list owner is frank.griffin@selectbs.com and the poster is ftg@roadrunner.com . It seems to me that this ought to be WebData Project Discussion <webdata@ftgme2.griffin.selectbs.com> .
Did I miss a step ? What can I do to correct this ?
Addenda: I set up a new list on the same host with ftg@roadrunner.com as the owner and as a subscriber, and sent a post from ftg@roadrunner.com to the list.
When ftg@roadrunner.com received the copy of the post, the headers were:
From: Frank Griffin <ftg@roadrunner.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20071128 SeaMonkey/1.1.8 MIME-Version: 1.0 To: frank.griffin@selectbs.com Subject: [Ftgtest] test X-BeenThere: ftgtest@ftgme2.griffin.selectbs.com X-Mailman-Version: 2.1.9 Precedence: list Reply-To: frank.griffin@selectbs.com List-Id: <ftgtest.ftgme2.griffin.selectbs.com> List-Unsubscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/ftgtest>, <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=unsubscribe> List-Archive: <http://ftgme2.griffin.selectbs.com/pipermail/ftgtest> List-Post: <mailto:ftgtest@ftgme2.griffin.selectbs.com> List-Help: <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=help> List-Subscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/ftgtest>, <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=subscribe> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: frank.griffin@selectbs.com Sender: Frank Griffin <ftg314159@gmail.com>
The interesting thing is that the "frank.griffin@selectbs.com" address, while the administrator address for the mailman server which previously owned all of the transported lists on the original host, doesn't figure *anywhere* in the definition of the ftgtest list on the new host. The ftgtest list has a null description, which is why its references here aren't of the form "xxxxx<frank.griffin@selectbs.com>" as in the previous example.
Frank Griffin wrote:
Addenda: I set up a new list on the same host with ftg@roadrunner.com as the owner and as a subscriber, and sent a post from ftg@roadrunner.com to the list.
When ftg@roadrunner.com received the copy of the post, the headers were:
From: Frank Griffin <ftg@roadrunner.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20071128 SeaMonkey/1.1.8 MIME-Version: 1.0 To: frank.griffin@selectbs.com
???
Subject: [Ftgtest] test X-BeenThere: ftgtest@ftgme2.griffin.selectbs.com X-Mailman-Version: 2.1.9 Precedence: list Reply-To: frank.griffin@selectbs.com List-Id: <ftgtest.ftgme2.griffin.selectbs.com> List-Unsubscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/ftgtest>, <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=unsubscribe> List-Archive: <http://ftgme2.griffin.selectbs.com/pipermail/ftgtest> List-Post: <mailto:ftgtest@ftgme2.griffin.selectbs.com> List-Help: <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=help> List-Subscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/ftgtest>, <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=subscribe> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: frank.griffin@selectbs.com Sender: Frank Griffin <ftg314159@gmail.com>
In a standard Mailman, Errors-To: and Sender: would both be <ftgtest-bounces@ftgme2.griffin.selectbs.com> as would the envelope sender (possibly visible in a Return-Path: header).
The interesting thing is that the "frank.griffin@selectbs.com" address, while the administrator address for the mailman server which previously owned all of the transported lists on the original host, doesn't figure *anywhere* in the definition of the ftgtest list on the new host.
Why is it the To: address of the post?
What mailman is this? I know it claims to be 2.1.9, but where does it come from and in what ways does it differ from the Mailman project's 2.1.9 distribution?
The ftgtest list has a null description, which is why its references here aren't of the form "xxxxx<frank.griffin@selectbs.com>" as in the previous example.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
Frank Griffin wrote:
Addenda: I set up a new list on the same host with ftg@roadrunner.com as the owner and as a subscriber, and sent a post from ftg@roadrunner.com to the list.
When ftg@roadrunner.com received the copy of the post, the headers were:
From: Frank Griffin <ftg@roadrunner.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20071128 SeaMonkey/1.1.8 MIME-Version: 1.0 To: frank.griffin@selectbs.com
???
Yeah :-)
Subject: [Ftgtest] test X-BeenThere: ftgtest@ftgme2.griffin.selectbs.com X-Mailman-Version: 2.1.9 Precedence: list Reply-To: frank.griffin@selectbs.com List-Id: <ftgtest.ftgme2.griffin.selectbs.com> List-Unsubscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/ftgtest>, <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=unsubscribe> List-Archive: <http://ftgme2.griffin.selectbs.com/pipermail/ftgtest> List-Post: <mailto:ftgtest@ftgme2.griffin.selectbs.com> List-Help: <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=help> List-Subscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/ftgtest>, <mailto:ftgtest-request@ftgme2.griffin.selectbs.com?subject=subscribe> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: frank.griffin@selectbs.com Sender: Frank Griffin <ftg314159@gmail.com>
In a standard Mailman, Errors-To: and Sender: would both be <ftgtest-bounces@ftgme2.griffin.selectbs.com> as would the envelope sender (possibly visible in a Return-Path: header).
The interesting thing is that the "frank.griffin@selectbs.com" address, while the administrator address for the mailman server which previously owned all of the transported lists on the original host, doesn't figure *anywhere* in the definition of the ftgtest list on the new host.
Why is it the To: address of the post?
You've got me. I'm assuming that all this stems from some anomaly in the upgrade that was done of the files copied from the old system.
What mailman is this? I know it claims to be 2.1.9, but where does it come from and in what ways does it differ from the Mailman project's 2.1.9 distribution?
It's the Mandriva Cooker (2008.1 RC2) rpm : mailman-2.1.9-4mdv2008.1.i586.rpm
Frank Griffin wrote:
Mark Sapiro wrote:
Why is it the To: address of the post?
You've got me. I'm assuming that all this stems from some anomaly in the upgrade that was done of the files copied from the old system.
It is clear to me that whatever Mailman this is, it is not the Mailman I know. I think that is the underlying problem.
What mailman is this? I know it claims to be 2.1.9, but where does it come from and in what ways does it differ from the Mailman project's 2.1.9 distribution?
It's the Mandriva Cooker (2008.1 RC2) rpm : mailman-2.1.9-4mdv2008.1.i586.rpm
It appears non-standard in at least a few obvious ways, and I have no way to know in what others. You might find answers using Mandriva support resources
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
It appears non-standard in at least a few obvious ways, and I have no way to know in what others. You might find answers using Mandriva support resources
I did before posting here, but found no bug reports there or on Google which resembled this in any way.
There's always the fallback of getting the source package and examining it, which I will do if I have to (my RPM skills are somewhat limited), but I can't imagine that nobody using Mandriva's package would have noticed this and reported it. I can't help but think that it's more probable that this is a porting issue.
I can, and will, do one more test short of pulling apart the RPM. I'll do a fresh Mandriva install, create a list, and see if the behavior is the same. If it isn't, that should point the finger at the porting.
Mark Sapiro writes:
It appears non-standard in at least a few obvious ways, and I have no way to know in what others. You might find answers using Mandriva support resources
Mandrake always prided itself on the excellence of its user and admin interfaces, and did not hesitate to change fundamental behaviors. In my experience it was at least as bad as cPanel or Plesk in that sense.
Dunno about Mandriva. YMMV ...
Frank Griffin wrote:
Thanks for the reply. I've recently gotten back to this and wanted to post my results.
In addition to the steps you posted, I needed to run "withlist -a -l -r fix_url".
However, I have one anomaly that still needs resolution:
All of the lists I moved had the option set to reply to the list (yes, I know), and they show up this way when I display them in the admin UI. However, when I post to them, the post mails arrive with a reply-to of the poster and the list description linked to an email address of the list owner, e.g.
List-Id: WebData Project Discussion <webdata.ftgme2.griffin.selectbs.com> List-Unsubscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/webdata>, <mailto:webdata-request@ftgme2.griffin.selectbs.com?subject=unsubscribe> List-Archive: <http://ftgme2.griffin.selectbs.com/pipermail/webdata> List-Post: <mailto:webdata@ftgme2.griffin.selectbs.com> List-Help: <mailto:webdata-request@ftgme2.griffin.selectbs.com?subject=help> List-Subscribe: <http://ftgme2.griffin.selectbs.com/mailman/listinfo/webdata>, <mailto:webdata-request@ftgme2.griffin.selectbs.com?subject=subscribe> Reply-To: ftg@roadrunner.com, WebData Project Discussion <frank.griffin@selectbs.com>
In this case, the list owner is frank.griffin@selectbs.com and the poster is ftg@roadrunner.com . It seems to me that this ought to be WebData Project Discussion <webdata@ftgme2.griffin.selectbs.com> .
Did I miss a step ? What can I do to correct this ?
I'm not sure what's happening in total, but the 'ftg@roadrunner.com' is your own Reply-To: from the incoming post. The default is to add any additional reply-to address to that. If you want it removed, you need to set first_strip_reply_to to Yes.
As far as the rest of it is concerned, I don't understand what's happening. If reply_goes_to_list is set to This list, what is put in Reply-To: should just be the list posting address <webdata@ftgme2.griffin.selectbs.com>. I don't know where the list description and owner address are coming from.
Is there anything in reply_to_address (although it doesn't seem that this could do what you see anyway)?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
I'm not sure what's happening in total, but the 'ftg@roadrunner.com' is your own Reply-To: from the incoming post. The default is to add any additional reply-to address to that. If you want it removed, you need to set first_strip_reply_to to Yes.
As far as the rest of it is concerned, I don't understand what's happening. If reply_goes_to_list is set to This list, what is put in Reply-To: should just be the list posting address <webdata@ftgme2.griffin.selectbs.com>. I don't know where the list description and owner address are coming from.
Is there anything in reply_to_address (although it doesn't seem that this could do what you see anyway)?
It wasn't the addition of ftg@roadrunner.com in the reply-to that made me curious (I can see that this list does that as well), but the fact that the second entry has the correct description but the wrong address. Referencing my later post, it seems to be getting the address from somewhere external to the actual list profile.
Where exactly is the authoritative location for these settings ? I'll print and post them.
Nothing shows in the UI for an explicit reply-to address, and the option to use one is not checked.
Frank Griffin wrote:
It wasn't the addition of ftg@roadrunner.com in the reply-to that made me curious (I can see that this list does that as well), but the fact that the second entry has the correct description but the wrong address. Referencing my later post, it seems to be getting the address from somewhere external to the actual list profile.
Where exactly is the authoritative location for these settings ? I'll print and post them.
In a standard Mailman, I trust the web GUI to display the correct thing in the three settings under "Reply-To: header munging" on the General Options page.
If you want to see what's behind that, you can do
bin/dumpdb lists/webdata/config.pck
and look at the actual values of the three attributes first_strip_reply_to, reply_goes_to_list and reply_to_address.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
Frank Griffin wrote:
It wasn't the addition of ftg@roadrunner.com in the reply-to that made me curious (I can see that this list does that as well), but the fact that the second entry has the correct description but the wrong address. Referencing my later post, it seems to be getting the address
from somewhere external to the actual list profile.
Where exactly is the authoritative location for these settings ? I'll print and post them.
In a standard Mailman, I trust the web GUI to display the correct thing in the three settings under "Reply-To: header munging" on the General Options page.
If you want to see what's behind that, you can do
bin/dumpdb lists/webdata/config.pck
and look at the actual values of the three attributes first_strip_reply_to, reply_goes_to_list and reply_to_address.
For the webdata list (the ported one):
'first_strip_reply_to': False,
'reply_goes_to_list': 1,
'reply_to_address': '',Outside of the 'passwords', 'members', and 'users' keys, the following keys contain 'frank.griffin@selectbs.com':
'owner': ['frank.griffin@selectbs.com'],That's it.
For the ftgtest list (created on the new host):
'first_strip_reply_to': 1,
'reply_goes_to_list': 1,
'reply_to_address': '',and no mention whatever of the address 'frank.griffin@selectbs.com'.
The mailman list contains:
'first_strip_reply_to': False,
'reply_goes_to_list': 0,
'reply_to_address': '',FWIW, there was no lists/mailman directory in the copied files.
Frank Griffin wrote:
Mark Sapiro wrote:
If you want to see what's behind that, you can do
bin/dumpdb lists/webdata/config.pck
and look at the actual values of the three attributes first_strip_reply_to, reply_goes_to_list and reply_to_address.
For the webdata list (the ported one):
'first_strip_reply_to': False, 'reply_goes_to_list': 1, 'reply_to_address': '',
Outside of the 'passwords', 'members', and 'users' keys, the following keys contain 'frank.griffin@selectbs.com':
'owner': ['frank.griffin@selectbs.com'],
Which in my Mailman says add the list posting address to the Reply-To:, if any, in the incoming message. Since in your case, it seems to be adding
list description <owner email>
This must be some Mandriva 'feature' about which I know nothing.
For the ftgtest list (created on the new host):
'first_strip_reply_to': 1, 'reply_goes_to_list': 1, 'reply_to_address': '',
and no mention whatever of the address 'frank.griffin@selectbs.com'.
Is it possible that you 'resent' an existing mail to the ftgtest list which had that To: header in it?
The mailman list contains:
'first_strip_reply_to': False, 'reply_goes_to_list': 0, 'reply_to_address': '',
FWIW, there was no lists/mailman directory in the copied files.
And this just says 'no Reply-To munging', and it is normal that in moving lists from another installation that you wouldn't move the mailman list (or in the case of moving from 2.0.x, wouldn't even have one to move). This list was probably installed by the rpm.
There are other anomalies that I can't explain. Normally, in standard Mailman, the RFC 2369 List- headers will follow the Reply-To: if any. That was true of the ftgtest post you posted earlier, but not the webdata post.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Frank Griffin -
Mark Sapiro -
Stephen J. Turnbull