Re: [Mailman-Users] Is there tool convert archived files/directories into a mbox file?
On 05/10/2013 06:35 AM, Xie, Wei wrote:
Here we have one customer to request migrating all old archived files of mailing list ‘kinejapan’ under directory /usr/local/mailman/archives/private/kinejapan prior to 2009 (as attached file) to Yale university. Yale university IT guys needs us to create a mbox file for these old archived file to be easy for them to run command ‘archdo migration.
I check there is no file kinejapan.mbox under directory /usr/local/mailman/archives/private/kinejapan.mbox to match these old archived files as attached. This mailing list was maintained by our pre-mailman admin, who passed away 2011, so I do not know why the mbox file was missing.
If ARCHIVE_TO_MBOX is set to 0 in mm_cfg.py, Mailman will not create/update LISTNAME.mbox/LISTNAME.mbox files.
Do you know whether there is a tool/method to convert these old archived files/directories as attached into a mbox file?
Lots of information is lost if there is no .mbox file.
You can do as well as anything with
cat /usr/local/mailman/archives/private/kinejapan/*.txt > kinejapan.mbox
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark,
If ARCHIVE_TO_MBOX is set to 0 in mm_cfg.py, Mailman will not create/update LISTNAME.mbox/LISTNAME.mbox files.
In the file mm_cfg.py, there is no ARCHIVE_TO_MBOX setting. There are ONLY following two parameters related to archive.
ARCHIVE_HTML_SANITIZER = 1 DEFAULT_ARCHIVE = Off
In the file Defaults.py, I see ARCHIVE_TO_MBOX = 2. Also, archiving for other mailing lists work well including creating/updating LISTNAME.mbox/LISTNAME.mbox file.
The archiving for this list 'kinejapan' does not work well. Last June we migrated this list from ListProc to mailman. And Archiving Options were enabled and it is supposed archiving has been working since last June. But two weeks ago we just found archiving did not really work.
- under /usr/local/mailman/archives/private/kinejapan, there was no new archived files/directories;
- there was no file kinejapan.mbox/kinejapan.mbox
Lots of information is lost if there is no .mbox file.
You can do as well as anything with
cat /usr/local/mailman/archives/private/kinejapan/*.txt > kinejapan.mbox
This works for us.
Thanks,
Carl
-----Original Message----- From: Mark Sapiro [mailto:mark@msapiro.net] Sent: Friday, May 10, 2013 11:37 AM To: Xie, Wei Cc: mailman-users@python.org Subject: Re: Is there tool convert archived files/directories into a mbox file?
On 05/10/2013 06:35 AM, Xie, Wei wrote:
Here we have one customer to request migrating all old archived files of mailing list 'kinejapan' under directory /usr/local/mailman/archives/private/kinejapan prior to 2009 (as attached file) to Yale university. Yale university IT guys needs us to create a mbox file for these old archived file to be easy for them to run command 'archdo migration.
I check there is no file kinejapan.mbox under directory /usr/local/mailman/archives/private/kinejapan.mbox to match these old archived files as attached. This mailing list was maintained by our pre-mailman admin, who passed away 2011, so I do not know why the mbox file was missing.
If ARCHIVE_TO_MBOX is set to 0 in mm_cfg.py, Mailman will not create/update LISTNAME.mbox/LISTNAME.mbox files.
Do you know whether there is a tool/method to convert these old archived files/directories as attached into a mbox file?
Lots of information is lost if there is no .mbox file.
You can do as well as anything with
cat /usr/local/mailman/archives/private/kinejapan/*.txt > kinejapan.mbox
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/10/2013 09:18 AM, Xie, Wei wrote:
In the file Defaults.py, I see ARCHIVE_TO_MBOX = 2.
That is the normal default meaning archive to moth pipermail and mbox.
Also, archiving for other mailing lists work well including creating/updating LISTNAME.mbox/LISTNAME.mbox file.
The archiving for this list 'kinejapan' does not work well. Last June we migrated this list from ListProc to mailman. And Archiving Options were enabled and it is supposed archiving has been working since last June. But two weeks ago we just found archiving did not really work.
- under /usr/local/mailman/archives/private/kinejapan, there was no new archived files/directories;
- there was no file kinejapan.mbox/kinejapan.mbox
I suspect a permissions issue. I suspect you will find a lot of 'shunting' and traceback messages in Mailman's 'error' log and a lot of posts in Mailman's shunt queue.
If this is the case, fixing the underlying error and then running bin/unshunt will archive the messages.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 05/10/2013 01:54 PM, Mark Sapiro wrote:
On 05/10/2013 09:18 AM, Xie, Wei wrote:
In the file Defaults.py, I see ARCHIVE_TO_MBOX = 2.
That is the normal default meaning archive to moth pipermail and mbox.
s/moth/both/
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I'm going to be migrating about 400 lists with 25K subscribers from ListProc to Mailman. I'm looking for whatever tools I can find to aid in the conversion process. Of particular concern is the creation of the replacement list in Mailman with a reasonable mapping of the ListProc settings into equivalent Mailman features, and of course the conversion of the subscribers. Archive conversion would be nice too, but is a much lower priority.
In my searching so far I've found a few bits and pieces, but nothing comprehensive. So before I get out the hammer and saw, I just want to make sure I'm not missing a good resource for doing this.
...BC
-- =====================================[ Bill.Costa@unh.edu ]== Bill Costa 1 Leavitt Lane UNH IT -- 1st Floor University of New Hampshire Durham, NH 03824 USA Voice: +1-603-862-3056 No good deed... Goes unpunished. ===========================[ http://pubpages.unh.edu/~wfc ]==
On 03/02/2015 07:24 AM, Bill.Costa@unh.edu wrote:
I'm going to be migrating about 400 lists with 25K subscribers from ListProc to Mailman. I'm looking for whatever tools I can find to aid in the conversion process. Of particular concern is the creation of the replacement list in Mailman with a reasonable mapping of the ListProc settings into equivalent Mailman features, and of course the conversion of the subscribers. Archive conversion would be nice too, but is a much lower priority.
I know nothing of the internals of ListProc, so I can't help directly, but there are a couple of Majordomo to Mailman conversion scripts (written in perl) in Mailman's contrib directory <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/files/head:/contrib/> that might give you some ideas.
Archives can be imported to Mailman using Mailman's bin/arch tool, but the messages need to be in a *nix mbox format with '^From ' separators. If they aren't already in that format, the task is to create it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro <mark@msapiro.net> recently posted, in part...
I know nothing of the internals of ListProc, so I can't help directly,
No problem there -- ListProc I know, warts and all.
but there are a couple of Majordomo to Mailman conversion scripts (written in perl) in Mailman's contrib directory <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/files/head:/contrib/> that might give you some ideas.
Definitely -- Perl is my native language. Thanks for the pointer.
Archives can be imported to Mailman using Mailman's bin/arch tool, but the messages need to be in a *nix mbox format...
That should be no problem.
...BC
-- =====================================[ Bill.Costa@unh.edu ]== Bill Costa 1 Leavitt Lane UNH IT -- 1st Floor University of New Hampshire Durham, NH 03824 USA Voice: +1-603-862-3056 No good deed... Goes unpunished. ===========================[ http://pubpages.unh.edu/~wfc ]==
A few years ago, we switched from ListProc to Mailman. We didn't have as extensive a set of lists as do you however.
Remember that communications with the list owners is extremely important.
What we did was: 0. Have a firm timeframe to complete the change over. lists are going to be named the same, simple create a list in Mailman
- Try to weed out as many lists as you can that may not be used anymore.
- Set up Mailman in parallel with ListProc.
- Create new lists in Mailman to replace the lists in ListProc (we changed our naming convention so there wasn't a 1 to 1 move). If your
for each list to be moved from ListProc. 4. List owners of the Mailman lists should be the same as ListProc lists. 5. Communicate with the list owners of the ListProc lists and let them know how to access and configure the list on Mailman. 6. Give list owners time to make their changes (e.g. add moderators, set moderation, etc.) and test the list. 7. After list owner testing, populate the list (or have the list owner do it) membership. 8. Communicate with the list owners for a cutover time (either individually or all at once) to do any file conversions like archives, etc. 9. Monitor the process carefully. 10. Remember that firm timeframe, well some list owners are still going to miss it. Be prepared.
Overall, the process went very smoothly for us. We did not convert the archives. We did change how we populated several of our lists though which was a bit of a challange (we went from populating via a static mechanism once per year to dynamic LDAP populating).
Chris
On 3/2/2015 7:24 AM, Bill.Costa@unh.edu wrote:
I'm going to be migrating about 400 lists with 25K subscribers from ListProc to Mailman. I'm looking for whatever tools I can find to aid in the conversion process. Of particular concern is the creation of the replacement list in Mailman with a reasonable mapping of the ListProc settings into equivalent Mailman features, and of course the conversion of the subscribers. Archive conversion would be nice too, but is a much lower priority.
In my searching so far I've found a few bits and pieces, but nothing comprehensive. So before I get out the hammer and saw, I just want to make sure I'm not missing a good resource for doing this.
...BC
Chris Nulk <cnulk@scu.edu> recently wrote, in part...
Remember that communications with the list owners is extremely important.
Indeed!
What we did was: 0. Have a firm timeframe to complete the change over.
Haven't picked the date yet -- nor have I told the users what is coming yet until I get Mailman installed, tested, and have done some test conversions. When that's all in place, you can bet your sweet bippy there will be a firm timeframe for the conversion!
- Try to weed out as many lists as you can that may not be used anymore.
Done. Contacted all list owners with no postings within the past year. One way or the other, all ended up being retired.
- Set up Mailman in parallel with ListProc.
Already have two separate list alias files so it will be easy to switch message delivery from ListProc to Mailman on a list-by-list basis.
- Create new lists in Mailman...
- List owners of the Mailman lists should be the same as ListProc lists.
- Communicate with the list owners of the ListProc lists and let them know how to access and configure the list on Mailman.
- Give list owners time to make their changes (e.g. add moderators, set moderation, etc.) and test the list.
- After list owner testing, populate the list (or have the list owner do it) membership.
My goal, which I did last time we did this (over 10 years ago from a homegrown system) is to have a script that does the conversion on a list-by-list basis. The list owner picks a date for their list to be 'down'. I do the conversion on that date, and bring their new list up and hand over the keys. This assumes that I will be able to do a good job of mapping the most important settings from ListProc to Mailman. If that doesn't turn out to be the case, your suggested approach of giving them some time to play with the Mailman list without subscribers may make sense. But for 80% of my list owners, once the list is initially created as either an 'announce', 'discussion', or 'moderated' list, they never mess with the configuration. So I'm optimistic I can map the list behavior automatically and have the new list working just like the old one, out of the gate.
- Communicate with the list owners for a cutover time (either individually or all at once) to do any file conversions like archives, etc.
In the last conversion I sent out an announcement to all the list owners telling why the conversion and the time-frame to contact me when they were ready to make the switch. That allowed me to make adjustments to the process as we went since my early adopters were better at both finding the bugs, and not panicking about same.
- Monitor the process carefully.
- Remember that firm timeframe, well some list owners are still going to miss it. Be prepared.
Indeed. Last time the deal was if I didn't hear from the list owner by such-and-such a date, the list would simply be retired. It was amazing how many people just let that happen rather than just replying to the nag message to say that they didn't want the list any more so just retire it now.
Overall, the process went very smoothly for us. We did not convert the archives.
We don't have many lists with archives. I probably will only convert if owners explicitly ask. I'm betting few will.
We did change how we populated several of our lists though which was a bit of a challange (we went from populating via a static mechanism once per year to dynamic LDAP populating).
I have a similar issue -- I maintain student announcement lists by college and class year, or even down to major code, with a weekly update from Registrar's records. I'll have to build that all over again for Mailman. Doesn't look like it should be too bad using the command line tools.
Thanks for the observations and advice.
...BC
-- =====================================[ Bill.Costa@unh.edu ]== Bill Costa 1 Leavitt Lane UNH IT -- 1st Floor University of New Hampshire Durham, NH 03824 USA Voice: +1-603-862-3056 No good deed... Goes unpunished. ===========================[ http://pubpages.unh.edu/~wfc ]==
On 3/2/2015 12:44 PM, Bill.Costa@unh.edu wrote:
Chris Nulk <cnulk@scu.edu> recently wrote, in part...
Remember that communications with the list owners is extremely important.
Indeed!
[snip]
We did change how we populated several of our lists though which was a bit of a challange (we went from populating via a static mechanism once per year to dynamic LDAP populating).
I have a similar issue -- I maintain student announcement lists by college and class year, or even down to major code, with a weekly update from Registrar's records. I'll have to build that all over again for Mailman. Doesn't look like it should be too bad using the command line tools.
If you can build LDAP queries to derive your list membership by college, class year, etc., you may want to look into using the LDAPMembership adapter. The only issues are list members cannot remove themselves from the list or change any user options. We use it to build four of our important "essential" communications lists for our community here (members are not allowed to remove themselves from the "essential" lists).
Thanks for the observations and advice.
...BC
No problem. Happy to help.
Chris
participants (4)
-
Bill.Costa@unh.edu
-
Chris Nulk
-
Mark Sapiro
-
Xie, Wei