Hello all, We have come up a few times where a list was created incorrectly. Normally, we catch the mistake, delete the list, and recreate it properly. However, there are times when the mistake is caught after having been in use. So, the delete and recreate is not available to use. I have a copy of Mark's clone_list command and it will allow us to rename lists. Which is great and not a problem for me. However, I am not the person normally involved with list creation, etc. Another group does it. Normally through the web interface which works well for them. Unless a mistake is made then they have to use the command line tools. Again, not a problem for me but that group has very limited knowledge of linux/unix and Mailman. I can easily train them to use the Mailman tools, however, ensuring they do the appropriate checks to avoid other problems is more difficult. Specifically, they can run the clone_list command but rebuilding the archives requires checking if the scrub_nondigest settings is No (this is explained in the clone_list help). What I have done is modify the clone_list command to combine the renaming/cloning of a list and include an option to rebuild the archives. However, the archives will not be rebuilt if the scrub_nondigest setting is Yes. Below my message is a diff of my changes to clone_list. While I try testing my changes on a test system, I would appreciate it if someone could take a look at my changes to ensure I am not off-base with my process. Thank you, Chris --------------- My diff of clone_list ----------- --- clone_list 2016-03-18 10:28:14.000000000 -0700 +++ clone_list_scu 2016-03-22 10:19:43.000000000 -0700 @@ -109,6 +109,10 @@ dest='archives', action='store_true', help="""\ Clone the archives of the old list. The default is an empty archive.""") + parser.add_argument('-b', '--rebuild_archives', + dest='rebuild_archives', action='store_true', + help="""\ +Rebuild the archives of the new list. Requires -a/--archives.""") parser.add_argument('-e', '--extra_files', dest='extra', action='store_true', help="""\ @@ -186,6 +190,8 @@ abort("%s doesn't appear to be a valid email address" % ns.owner) if ns.extra and not ns.clone_members: abort('-e/--extra_files requires -m/--members.') + if ns.rebuild_archives and not ns.archives: + abort('-b/--rebuild_archives requires -a/--archives.') if ns.verbose: print 'Getting %s list...' % ns.old_list ol = MailList(old_list, lock=False) @@ -268,5 +274,29 @@ (ns.old_list, ns.new_list)) copy_archives(old_list, new_list) + if ns.rebuild_archives: + if ns.verbose: + print 'Rebuilding %s archives...' % ns.new_list + if ol.scrub_nondigest: + abort('Scrub_nondigest is YES for %s. WILL NOT REBUILD ARCHIVES.' % ns.old_list) + else: + archcmd = os.path.join(os.path.dirname(sys.argv[0]), 'arch') + if not os.path.isfile(archcmd): + abort("""%s doesn't exist. +Am I installed in Mailman's bin/ directory?""" % archcmd) + rbld = subprocess.Popen([archcmd, + '--wipe', + new_list + ], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE + ) + so, se = rbld.communicate() + if rbld.returncode: + abort('unable to rebuild archives for %s\n%s' % (ns.new_list, se)) + # If there was stdout output, print it. It is probably aliases. + if so: + print so + if __name__ == '__main__': main()
On 03/22/2016 11:02 AM, Chris Nulk wrote:
Below my message is a diff of my changes to clone_list. While I try testing my changes on a test system, I would appreciate it if someone could take a look at my changes to ensure I am not off-base with my process.
I have looked at it and it looks fine to me.
I will try adding it to my script and doing a bit of testing, but I think it will be fine.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
CN> What I have done is modify the clone_list command to combine the CN> renaming/cloning of a list and include an option to rebuild the CN> archives. However, the archives will not be rebuilt if the CN> scrub_nondigest setting is Yes.
Sounds like a worthwhile enhancement.
MS> I will try adding it to my script and doing a bit of testing, but I MS> think it will be fine.
Does this mean it will be included in the next release of MM v2?
As for getting off on the right foot for a new list, as part of our new list application process...
http://lists.unh.edu/MM/NewList/
we have the new list owner pick an initial flavor for their list -- announce, discussion, moderated. The backend uses one of three MM list definition templates to apply the appropriate defaults for the given list type. I used this system for about 15 years with ListProc and it has always worked well. (I'd offer the share the code, but it is in Perl. It has extensive commenting, but would still probably take a fair amount of grokking and hacking to get it to work at a new site. It was never designed to distributed.)
BTW -- if anyone goes through the application process above to take a look, please use the list name 'bogus.list' so I can ignore/delete it.
-- Cordially, the UNH Mailing List System Admins Bill Costa, senior admin (603) 862-3056
Thanks for the info Bill.
I agree getting off on the right foot for a new list is important. We have a basic form users fill out which includes our taxonomy for placing the lists. The message the group gets has all the information. The problem is sometimes bits of the key taxonomy are left off.
Thanks, Chris
On 3/23/2016 7:56 AM, The Mailing List System Admin wrote:
CN> What I have done is modify the clone_list command to combine the CN> renaming/cloning of a list and include an option to rebuild the CN> archives. However, the archives will not be rebuilt if the CN> scrub_nondigest setting is Yes.
Sounds like a worthwhile enhancement.
MS> I will try adding it to my script and doing a bit of testing, but I MS> think it will be fine.
Does this mean it will be included in the next release of MM v2?
As for getting off on the right foot for a new list, as part of our new list application process...
http://lists.unh.edu/MM/NewList/
we have the new list owner pick an initial flavor for their list -- announce, discussion, moderated. The backend uses one of three MM list definition templates to apply the appropriate defaults for the given list type. I used this system for about 15 years with ListProc and it has always worked well. (I'd offer the share the code, but it is in Perl. It has extensive commenting, but would still probably take a fair amount of grokking and hacking to get it to work at a new site. It was never designed to distributed.)
BTW -- if anyone goes through the application process above to take a look, please use the list name 'bogus.list' so I can ignore/delete it.
On 3/22/2016 5:03 PM, Mark Sapiro wrote:
On 03/22/2016 11:02 AM, Chris Nulk wrote:
Below my message is a diff of my changes to clone_list. While I try testing my changes on a test system, I would appreciate it if someone could take a look at my changes to ensure I am not off-base with my process.
I have looked at it and it looks fine to me.
I will try adding it to my script and doing a bit of testing, but I think it will be fine.
Thanks for looking at it Mark.
I have done limited testing (4 member list with a few archived messages)
and it appears to be working. The archives were moved and rebuilt.
When I changed the scrub_nondigest setting to Yes, the list is renamed,
archives moved, and an error message that the archives were not rebuilt.
Based on my testing, I noticed the error message for not rebuilding the archives is a bit terse. For my target audience it needs to be a little more helpful. So, I changed the lines:
if ol.scrub_nondigest:
abort('Scrub_nondigest is YES for %s. WILL NOT REBUILD ARCHIVES.' % ns.old_list) else:
with
if ol.scrub_nondigest:
abort("""ARCHIVES NOT REBUILT.Scrub_nondigest for %s is set to Yes. To prevent losing any scrubbed attachments, you will need to rebuild and fix the listname links in the archives using another method than using bin/arch. """ % ns.old_list) else:
I will do a little more testing on some larger lists but it looks promising.
Thank you again Mark, Chris
On 03/23/2016 11:15 AM, Chris Nulk wrote:
I have done limited testing (4 member list with a few archived messages) and it appears to be working. The archives were moved and rebuilt. When I changed the scrub_nondigest setting to Yes, the list is renamed, archives moved, and an error message that the archives were not rebuilt.
Actually, is scrub_nondigest is Yes, bin/arch --wipe will work but there are lots of caveats. In this case, bin/arch --wipe will just not remove the 'attachments' directory. The issue is if scrub_nondigest was always Yes for this list, that's the right thing to do, but it it was No at some time in the past and changed to Yes, there are already two copies of each attachment from the No period (one scrubbed from the archive and one from the plain digest) and running bin/arch --wip will leave those and create third copies from the attachments in the mbox.
The more serious issue is if scrub_nondigest was Yes at some time in the past and is now No, bin/arch --wipe will lose the attachments from the Yes period.
In any case, for attachments scrubbed by scrub_nondigest, there are links to them in the message bodies in the mbox and the list names in those links will now be wrong, and bin/arch won't (can't) fix them.
This is why I chose in my script to kick the issue back to the user.
Based on my testing, I noticed the error message for not rebuilding the archives is a bit terse. For my target audience it needs to be a little more helpful. So, I changed the lines:
if ol.scrub_nondigest: abort('Scrub_nondigest is YES for %s. WILL NOT REBUILDARCHIVES.' % ns.old_list) else:
with
if ol.scrub_nondigest: abort("""ARCHIVES NOT REBUILT.Scrub_nondigest for %s is set to Yes. To prevent losing any scrubbed attachments, you will need to rebuild and fix the listname links in the archives using another method than using bin/arch. """ % ns.old_list) else:
That much seems good, but as I note above it's a bit more complicated if the scrub_nondigest setting has changed over time.
I will do a little more testing on some larger lists but it looks promising.
OK
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 3/23/2016 6:42 PM, Mark Sapiro wrote:
On 03/23/2016 11:15 AM, Chris Nulk wrote:
I have done limited testing (4 member list with a few archived messages) and it appears to be working. The archives were moved and rebuilt. When I changed the scrub_nondigest setting to Yes, the list is renamed, archives moved, and an error message that the archives were not rebuilt.
Actually, is scrub_nondigest is Yes, bin/arch --wipe will work but there are lots of caveats. In this case, bin/arch --wipe will just not remove the 'attachments' directory. The issue is if scrub_nondigest was always Yes for this list, that's the right thing to do, but it it was No at some time in the past and changed to Yes, there are already two copies of each attachment from the No period (one scrubbed from the archive and one from the plain digest) and running bin/arch --wip will leave those and create third copies from the attachments in the mbox.
The more serious issue is if scrub_nondigest was Yes at some time in the past and is now No, bin/arch --wipe will lose the attachments from the Yes period.
In any case, for attachments scrubbed by scrub_nondigest, there are links to them in the message bodies in the mbox and the list names in those links will now be wrong, and bin/arch won't (can't) fix them.
This is why I chose in my script to kick the issue back to the user.
I figured it was a bit more complicated. So, there is still a possibility of losing data if scrub_nondigest is No since the list doesn't maintain a history of previous settings.
Based on my testing, I noticed the error message for not rebuilding the archives is a bit terse. For my target audience it needs to be a little more helpful. So, I changed the lines:
if ol.scrub_nondigest: abort('Scrub_nondigest is YES for %s. WILL NOT REBUILDARCHIVES.' % ns.old_list) else:
with
if ol.scrub_nondigest: abort("""ARCHIVES NOT REBUILT.Scrub_nondigest for %s is set to Yes. To prevent losing any scrubbed attachments, you will need to rebuild and fix the listname links in the archives using another method than using bin/arch. """ % ns.old_list) else:
That much seems good, but as I note above it's a bit more complicated if the scrub_nondigest setting has changed over time.
I did a little searching around our lists. It seems the attachments directory is always created and populated whether or not scrub_nondigest was ever set to Yes. So I can't really use the existence of the attachments directory or the lack of it being populated as a check for if scrub_nondigest was ever set to Yes.
Is the anything that can be checked for that would indicate scrub_nondigest was Yes at any point in the past? Obviously, if it is currently Yes it doesn't matter.
Lacking the ability to check, I suppose a warning should be generated prior to running the bin/arch. In addition, the '-b/--rebuild_archives' option should default to false unless there is a positive indication the user wants to run it ('-b y / --rebuild_archives=Yes'). The default for the rebuild_archives option would be No.
If there is any way to tell if scrub_nondigest has been set to Yes in the list's past, please let me know. I am going to add the warning and try making the changes to the rebuild_archives options. I will post the diff from the Mark's original clone_list.
Thanks Mark, Chris
On 03/24/2016 09:17 AM, Chris Nulk wrote:
I did a little searching around our lists. It seems the attachments directory is always created and populated whether or not scrub_nondigest was ever set to Yes. So I can't really use the existence of the attachments directory or the lack of it being populated as a check for if scrub_nondigest was ever set to Yes.
No, but here's what you can look for.
If scrub_nondigest is True, the message will be scrubbed during processing before any delivery and before the message is added to the cumulative mbox. Thus there will be one copy of each scrubbed attachment in the attachments directory and a link in the message body and the scrubbed attachment will not be in the cumulative mbox.
If scrub_nondigest is False, archiving will scrub the message and any scrubbed attachments will be in the attachments directory and replaced by links in the archived message, and the attachment will be in the cumulative mbox. Additionally, if the list is digestable, digest processing will also scrub the message for the plain text digest even if there are no plain digest subscribers or even no MIME digest subscribers either. This will create second copies of each attachment in the attachments directory, but not until the digest is actually created.
Is the anything that can be checked for that would indicate scrub_nondigest was Yes at any point in the past? Obviously, if it is currently Yes it doesn't matter.
Not 100% reliably, but if there are periods of time when the attachments (sub-)directories contain two copies of each attachment, you know scrub_nondigest was No at that point, and the absence of any duplicates may say scrub_nondigest was Yes at that point or it may say that this archive was imported or previously processed with bin/arch --wipe which will not replace the duplicate attachments.
Lacking the ability to check, I suppose a warning should be generated prior to running the bin/arch. In addition, the '-b/--rebuild_archives' option should default to false unless there is a positive indication the user wants to run it ('-b y / --rebuild_archives=Yes'). The default for the rebuild_archives option would be No.
If there is any way to tell if scrub_nondigest has been set to Yes in the list's past, please let me know. I am going to add the warning and try making the changes to the rebuild_archives options. I will post the diff from the Mark's original clone_list.
It just occurred to me that there is a way. Check the lists cumulative mbox for messages which are single part text/plain and have content somewhere in the body matching the regexp
'\nAn? .* was scrubbed\.\.\.\n'
This would indicate the message was scrubbed before being archived (or possibly is a message quoting something from a plain text digest, there's always a gotcha) and would probably mean that at the time of that message, scrub_nondigest was Yes.
It's actually OK to run bin/arch --wipe if scrub_nondigest is Yes. The two 'bad' things that happen here are if scrub_nondigest was No in the past, there will be extra, unreferenced copies of attachments, and for the periods when scrub_nondigest was Yes, the links to scrubbed attachments will still have the wrong listname.
The loss of data problem is if scrub_nondigest is No and was Yes in the past, the attachments that were scrubbed when it was Yes will be lost.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 3/24/2016 11:49 AM, Mark Sapiro wrote:
On 03/24/2016 09:17 AM, Chris Nulk wrote:
I did a little searching around our lists. It seems the attachments directory is always created and populated whether or not scrub_nondigest was ever set to Yes. So I can't really use the existence of the attachments directory or the lack of it being populated as a check for if scrub_nondigest was ever set to Yes.
No, but here's what you can look for.
If scrub_nondigest is True, the message will be scrubbed during processing before any delivery and before the message is added to the cumulative mbox. Thus there will be one copy of each scrubbed attachment in the attachments directory and a link in the message body and the scrubbed attachment will not be in the cumulative mbox.
[SNIPPED a big chunk]
The loss of data problem is if scrub_nondigest is No and was Yes in the past, the attachments that were scrubbed when it was Yes will be lost.
Thanks for the information Mark. It is a messy situation. I decided to simplify the changes I made to clone_list. 1. The -b/--rebuild_archives option now requires a Yes/No/? answer (don't know if I did it right in the parser) 2. If -b/--rebuild_archives is: a. specified on the command line but without Yes/No/?, '?' is assumed, b. '?', program prints warning message and exits, c. 'No', program proceeds with list cloning and does NOT rebuild the archives, d. 'Yes', program proceeds with list cloning and DOES rebuild the archives (no warnings). Hopefully, the changes will catch casual mistakes. Below is the diff from Mark's original clone_list. Please let me know if there are any improvements to be made. Thanks, Chris ------------------- diff file ------------------------ --- clone_list 2016-03-18 10:28:14.000000000 -0700 +++ clone_list_scu 2016-03-29 11:23:22.000000000 -0700 @@ -109,6 +109,17 @@ dest='archives', action='store_true', help="""\ Clone the archives of the old list. The default is an empty archive.""") + parser.add_argument('-b', '--rebuild_archives', + dest='rebuild_archives', nargs='?', + const='?', default='No', + help="""\ +Rebuild the archives of the new list. Requires -a/--archives. + ***** WARNING ***** +If scrub_nondigest was 'Yes' at any time in a list's past, +rebuilding the archives will lose the scrubbed attachments from the +'Yes' period. Please review the settings and archives for the list +prior to rebuilding the archives. At a minimum, make backups of the +list and its archives.""") parser.add_argument('-e', '--extra_files', dest='extra', action='store_true', help="""\ @@ -186,6 +197,25 @@ abort("%s doesn't appear to be a valid email address" % ns.owner) if ns.extra and not ns.clone_members: abort('-e/--extra_files requires -m/--members.') + + if ns.rebuild_archives.lower() not in set(['no', 'n', 'yes', 'y', '?']): + abort('%s is not valid for -b/--rebuild_archives. Please answer Yes or No.' % ns.rebuild_archives) + rebuild_archives = ns.rebuild_archives[0].lower() + if rebuild_archives == '?': + abort(""" +Specifying rebuild_archives requires an affirmative indication for +%s's archives to be rebuilt. + + ***** WARNING ***** +If scrub_nondigest was 'Yes' at any time in %s's past, +rebuilding the archives will lose the scrubbed attachments from the +'Yes' period. Please review the settings and archives for %s +prior to rebuilding the archives. At a minimum, make backups of the +list and its archives.""" % (old_list, old_list, old_list)) + + if rebuild_archives == 'y' and not ns.archives: + abort('-b/--rebuild_archives requires -a/--archives.') + if ns.verbose: print 'Getting %s list...' % ns.old_list ol = MailList(old_list, lock=False) @@ -268,5 +298,26 @@ (ns.old_list, ns.new_list)) copy_archives(old_list, new_list) + if rebuild_archives == 'y': + if ns.verbose: + print 'Rebuilding %s archives...' % ns.new_list + archcmd = os.path.join(os.path.dirname(sys.argv[0]), 'arch') + if not os.path.isfile(archcmd): + abort("""%s doesn't exist. +Am I installed in Mailman's bin/ directory?""" % archcmd) + rbld = subprocess.Popen([archcmd, + '--wipe', + new_list + ], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE + ) + so, se = rbld.communicate() + if rbld.returncode: + abort('unable to rebuild archives for %s\n%s' % (ns.new_list, se)) + # If there was stdout output, print it. It is probably aliases. + if so: + print so + if __name__ == '__main__': main()
On Tue, Mar 22, 2016 at 11:02:10AM -0700, Chris Nulk wrote:
I have a copy of Mark's clone_list command and it will allow us to rename lists. Which is great and not a problem for me. However, I am not the person normally involved with list creation, etc. Another group does it. Normally through the web interface which works well for them. Unless a mistake is made then they have to use the command line tools. Again, not a problem for me but that group has very limited knowledge of linux/unix and Mailman.
If they can be assumed to use old list and new list as parameters (although I could do a check for each…)
https://gist.github.com/adamamyl/6909815#file-rename-list may be useful
invoked as rename-list old-list new-list
-- "If the media object to a judgment or sentencing decision, we suggest they focus their efforts on persuading the Government to rectify the legal and policy framework." -- Lords' Select Committee on Constitution: Eleventh Report
On 3/23/2016 10:39 AM, Adam McGreggor wrote:
On Tue, Mar 22, 2016 at 11:02:10AM -0700, Chris Nulk wrote:
I have a copy of Mark's clone_list command and it will allow us to rename lists. Which is great and not a problem for me. However, I am not the person normally involved with list creation, etc. Another group does it. Normally through the web interface which works well for them. Unless a mistake is made then they have to use the command line tools. Again, not a problem for me but that group has very limited knowledge of linux/unix and Mailman. If they can be assumed to use old list and new list as parameters (although I could do a check for each…)
https://gist.github.com/adamamyl/6909815#file-rename-listmay be useful
invoked as
rename-list old-list new-list
Thanks for the info Adam. I was planning on creating a script to do the renaming like your script. While looking at the Mailman FAQ's on renaming a list, I saw the link to Mark's clone_list. To make it easier for the group that would be using the script, I added in the bits to also copy the archives and rebuild them. I am just happy I don't have to write the bash script I was planning. The process has eighteen steps including notifying the list owner (important), doing the work, verifying everything, and administrative work. The new process using clone_list has seven steps most of which is administrative.
Thanks again, Chris
On Wed, Mar 23, 2016 at 11:25:21AM -0700, Chris Nulk wrote:
On 3/23/2016 10:39 AM, Adam McGreggor wrote:
https://gist.github.com/adamamyl/6909815#file-rename-listmay be useful
invoked as
rename-list old-list new-listThanks for the info Adam. I was planning on creating a script to do the renaming like your script. While looking at the Mailman FAQ's on renaming a list, I saw the link to Mark's clone_list. To make it easier for the group that would be using the script, I added in the bits to also copy the archives and rebuild them. I am just happy I don't have to write the bash script I was planning. The process has eighteen steps including notifying the list owner (important), doing the work, verifying everything, and administrative work. The new process using clone_list has seven steps most of which is administrative.
I wrote mine on the bits I needed; change the list name, swing archives, quick exim redirect/alias entry…
-- "You can't say that, because it's true." (unnamed Russian censor, to Malcom Muggeridge, 1933)
participants (5)
-
Adam McGreggor -
Adam McGreggor -
Chris Nulk -
Mark Sapiro -
The Mailing List System Admin