[Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3
Public bug reported:
We need a script, documentation, or other procedure to help people migrate from Mailman 2 to Mailman 3.
** Affects: mailman Importance: Undecided Status: New
** Tags: mailman3
** Changed in: mailman Status: New => Confirmed
** Changed in: mailman Importance: Undecided => High
** Changed in: mailman Importance: High => Critical
** Changed in: mailman Milestone: None => 3.0.0b2
** Changed in: mailman Assignee: (unassigned) => Richard Wackerbarth (wacky)
On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
Need a script to upgrade from MM2 to MM3 https://bugs.launchpad.net/bugs/965532
Here are some thoughts on a possible migration technique. I would request discussion and suggestions.
In particular, what about the idea of converting the configuration file to HTML as an intermediate file format? Selectable css could easily render it as a viewable report. It could still be edited by hand without too much difficulty.
Richard "Wacky" Wackerbarth - - - - - -
Steps to migrate from MM2 to MM3
1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should include testing to assure that things are ready to create new lists.
2) Translate list configurations
a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would populate configurations at the List level. It might also reformat selected parameters. In particular, various <option type="radio" > entries might be transformed into enumerations such as "Yes"/"No" or "Hidden"/"Private"/"Public" rather than numerical values. This would enhance readability.
b) TOOL3 would populate the World level with the MM2 defaults and recursively promote common values up the tree, leaving only those entries which would need to override their parent to derive the current value. Values which match the parent would be flagged. (The inheritance flag should be tri-state. "Differs from parent", "Same as parent", "Inherited from parent")
c) At this point, the user might edit some of the configurations and rerun TOOL3 adjusting the inheritance flag as appropriate.
d) Now, we begin translation to MM3 configuration options. For each MM3 option, TOOL4 computes the equivalent value from the MM2 values. Each computed value also gets the corresponding inheritance flag. Values that cannot be computed from the available information get the "Inherited from parent" flag. MM2 values used in computations are marked as "translated". e) After a chance to edit the MM3 configurations, TOOL5 would recompute inheritance flags, report any MM2 values that have not been translated and produce a copy of the configuration file simplified by removing all inherited entries.
f) After a final inspection TOOL7 would actually import the configurations, committing entries to the MM3 database.
3) For the migration of rosters, we should be able to do it one subscription at a time through a pipeline that permits pre- and post- hooks. A --dry-run option would be appropriate.
a) We can assume that each email address is a distinct person. The subscribers can utilize the UI to merge email addresses into a common persona.
b) We can also assume that each subscription overrides its parent in the Persona-->EMailAddress-->Subscription hierarchy. The individual users can use the UI to consolidate their selections.
Some additional thoughts:
All of the tools should be written in Python, hopefully in a dialect that is common to all of the versions supported by MM3. TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes as they come in. Similarly, in step 3, we can use TOOL1 and discard the list configuration information. TOOL2 can reformat the XML as HTML, thus making the input data into a viewable report. The inheritance flag would become a class attribute on the option. Would it make sense to go a step further and generate html forms and run a trivial http server on localhost? It might be easier to do this in django, but I think that requiring that level of installation is probably too much for the current situation.
On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:
On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
Need a script to upgrade from MM2 to MM3 https://bugs.launchpad.net/bugs/965532
Here are some thoughts on a possible migration technique. I would request discussion and suggestions.
In particular, what about the idea of converting the configuration file to HTML as an intermediate file format? Selectable css could easily render it as a viewable report. It could still be edited by hand without too much difficulty.
It's an interesting idea. As you observed, mm2 can export to XML, so it's not such a big stretch.
w>Steps to migrate from MM2 to MM3
- Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
include testing to assure that things are ready to create new lists.
Right, and it should be doable even while mm2 is still functional.
- Translate list configurations
a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would populate configurations at the List level. It might also reformat selected parameters. In particular, various <option type="radio" > entries might be transformed into enumerations such as "Yes"/"No" or "Hidden"/"Private"/"Public" rather than numerical values. This would enhance readability.
b) TOOL3 would populate the World level with the MM2 defaults and recursively promote common values up the tree, leaving only those entries which would need to override their parent to derive the current value. Values which match the parent would be flagged. (The inheritance flag should be tri-state. "Differs from parent", "Same as parent", "Inherited from parent")
c) At this point, the user might edit some of the configurations and rerun TOOL3 adjusting the inheritance flag as appropriate.
I think the trickiest part will be what to do about subscriber information. In mm2, this is always list-centric, but in mm3, you need to collate and globalize all the membership information into the user database. You can probably do the same kind of up-promotion there, but it would be from member->address->user. IOW, if you see an address subscribed to a mailing list with the same values across all those lists, put the preferences in the user. What happens if you see anne@example.com subscribed to three different lists with three different passwords? That's a tough one because there's no way to express that in mm3 (nor probably should there be).
So I think you will occasionally have to just resolve some conflicts by flipping a coin. In the case of passwords, perhaps you'd always make the user do a password reset.
d) Now, we begin translation to MM3 configuration options. For each MM3 option, TOOL4 computes the equivalent value from the MM2 values. Each computed value also gets the corresponding inheritance flag. Values that cannot be computed from the available information get the "Inherited from parent" flag. MM2 values used in computations are marked as "translated".
The user herself could probably write a script for this pipeline you're proposing, that would allow her to do bulk transformations of configuration variable.
e) After a chance to edit the MM3 configurations, TOOL5 would recompute inheritance flags, report any MM2 values that have not been translated and produce a copy of the configuration file simplified by removing all inherited entries.
f) After a final inspection TOOL7 would actually import the configurations, committing entries to the MM3 database.
- For the migration of rosters, we should be able to do it one subscription
at a time through a pipeline that permits pre- and post- hooks. A --dry-run option would be appropriate.
Almost definitely. The --dry-run step which would produce an output of those conflicts, and impossible situations. The user would then have a chance to re-edit the intermediate file so that the values can be better mapped to mm3. It's probably worth doing for both the rosters and list configurations.
a) We can assume that each email address is a distinct person. The subscribers can utilize the UI to merge email addresses into a common persona.
We'll probably need a "claim and merge" operation in the system. And a way to purge unclaimed addresses after a while.
b) We can also assume that each subscription overrides its parent in the Persona-->EMailAddress-->Subscription hierarchy. The individual users can use the UI to consolidate their selections.
A good challenge for the Postoriusians :)
All of the tools should be written in Python, hopefully in a dialect that is common to all of the versions supported by MM3.
+1. Today that would be 2.6 and 2.7.
TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes as they come in. Similarly, in step 3, we can use TOOL1 and discard the list configuration information.
TOOL2 can reformat the XML as HTML, thus making the input data into a viewable report. The inheritance flag would become a class attribute on the option. Would it make sense to go a step further and generate html forms and run a trivial http server on localhost? It might be easier to do this in django, but I think that requiring that level of installation is probably too much for the current situation.
It wouldn't be hard to do in standard Python. OTOH, I'm not sure we want to maintain a stack of html templates, forms, and form processing in the conversion tool.
It's sounding like this suite of conversion tools may need to be a separate sub-project.
-Barry
Many MM admins are NOT programmers. I could do the transforms in either Python or XSL.
Which would be more acceptable to our users who might want to customize the transform? In either case, the transforms would be kept in a separate file from the rest of the engine and pipeline so that "where to customize" is clear.
Richard
On Apr 10, 2012, at 3:46 PM, Barry Warsaw 965532@bugs.launchpad.net wrote:
On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:
On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
Need a script to upgrade from MM2 to MM3 https://bugs.launchpad.net/bugs/965532
Here are some thoughts on a possible migration technique. I would request discussion and suggestions.
In particular, what about the idea of converting the configuration file to HTML as an intermediate file format? Selectable css could easily render it as a viewable report. It could still be edited by hand without too much difficulty.
It's an interesting idea. As you observed, mm2 can export to XML, so it's not such a big stretch.
w>Steps to migrate from MM2 to MM3
- Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
include testing to assure that things are ready to create new lists.
Right, and it should be doable even while mm2 is still functional.
- Translate list configurations
a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would populate configurations at the List level. It might also reformat selected parameters. In particular, various <option type="radio" > entries might be transformed into enumerations such as "Yes"/"No" or "Hidden"/"Private"/"Public" rather than numerical values. This would enhance readability.
b) TOOL3 would populate the World level with the MM2 defaults and recursively promote common values up the tree, leaving only those entries which would need to override their parent to derive the current value. Values which match the parent would be flagged. (The inheritance flag should be tri-state. "Differs from parent", "Same as parent", "Inherited from parent")
c) At this point, the user might edit some of the configurations and rerun TOOL3 adjusting the inheritance flag as appropriate.
I think the trickiest part will be what to do about subscriber information. In mm2, this is always list-centric, but in mm3, you need to collate and globalize all the membership information into the user database. You can probably do the same kind of up-promotion there, but it would be from member->address->user. IOW, if you see an address subscribed to a mailing list with the same values across all those lists, put the preferences in the user. What happens if you see anne@example.com subscribed to three different lists with three different passwords? That's a tough one because there's no way to express that in mm3 (nor probably should there be).
So I think you will occasionally have to just resolve some conflicts by flipping a coin. In the case of passwords, perhaps you'd always make the user do a password reset.
d) Now, we begin translation to MM3 configuration options. For each MM3 option, TOOL4 computes the equivalent value from the MM2 values. Each computed value also gets the corresponding inheritance flag. Values that cannot be computed from the available information get the "Inherited from parent" flag. MM2 values used in computations are marked as "translated".
The user herself could probably write a script for this pipeline you're proposing, that would allow her to do bulk transformations of configuration variable.
e) After a chance to edit the MM3 configurations, TOOL5 would recompute inheritance flags, report any MM2 values that have not been translated and produce a copy of the configuration file simplified by removing all inherited entries.
f) After a final inspection TOOL7 would actually import the configurations, committing entries to the MM3 database.
- For the migration of rosters, we should be able to do it one subscription
at a time through a pipeline that permits pre- and post- hooks. A --dry-run option would be appropriate.
Almost definitely. The --dry-run step which would produce an output of those conflicts, and impossible situations. The user would then have a chance to re-edit the intermediate file so that the values can be better mapped to mm3. It's probably worth doing for both the rosters and list configurations.
a) We can assume that each email address is a distinct person. The subscribers can utilize the UI to merge email addresses into a common persona.
We'll probably need a "claim and merge" operation in the system. And a way to purge unclaimed addresses after a while.
b) We can also assume that each subscription overrides its parent in the Persona-->EMailAddress-->Subscription hierarchy. The individual users can use the UI to consolidate their selections.
A good challenge for the Postoriusians :)
All of the tools should be written in Python, hopefully in a dialect that is common to all of the versions supported by MM3.
+1. Today that would be 2.6 and 2.7.
TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes as they come in. Similarly, in step 3, we can use TOOL1 and discard the list configuration information.
TOOL2 can reformat the XML as HTML, thus making the input data into a viewable report. The inheritance flag would become a class attribute on the option. Would it make sense to go a step further and generate html forms and run a trivial http server on localhost? It might be easier to do this in django, but I think that requiring that level of installation is probably too much for the current situation.
It wouldn't be hard to do in standard Python. OTOH, I'm not sure we want to maintain a stack of html templates, forms, and form processing in the conversion tool.
It's sounding like this suite of conversion tools may need to be a separate sub-project.
-Barry
-- You received this bug notification because you are a bug assignee. https://bugs.launchpad.net/bugs/965532
Title: Need a script to upgrade from MM2 to MM3
Status in GNU Mailman: Confirmed
Bug description: We need a script, documentation, or other procedure to help people migrate from Mailman 2 to Mailman 3.
To manage notifications about this bug go to: https://bugs.launchpad.net/mailman/+bug/965532/+subscriptions
** Changed in: mailman Milestone: 3.0.0b2 => 3.0.0b3
** Changed in: mailman Milestone: 3.0.0b3 => 3.0.0b4
That is really important thing to make it work.
has any work on this taken place? is there documentation anywhere?
abompard has a branch, which I've slowly been working through.
** Changed in: mailman Milestone: 3.0.0b4 => None
Should this bug have had its milestone removed on the release of b4?
Any help needed? How much work do you envisage remains here? As a python coder and mailman admin, I wonder if I can help.
I can't tell if the code was merged or not. It says "abstain": https://code.launchpad.net/~abompard/mailman/import21/+merge/192146
On Sep 08, 2014, at 12:04 AM, Kẏra wrote:
I can't tell if the code was merged or not. It says "abstain": https://code.launchpad.net/~abompard/mailman/import21/+merge/192146
Yes, that branch was merged.
great, so this bug can be marked as resolved?
is this bug still unresolved despite the code merge?
On Feb 03, 2015, at 10:21 PM, Kẏra wrote:
is this bug still unresolved despite the code merge?
It probably needs more testing with real-world data.
This bug has been moved to the new gitlab repo here: https://gitlab.com/maxking/mailman/issues/4
This bug has been moved to the new gitlab repo here: https://gitlab.com/maxking/mailman/issues/5
Please ignore the above two comments, the new url on gitlab is : https://gitlab.com/mailman/mailman/issues/4
** Changed in: mailman Status: Confirmed => Fix Released
participants (8)
-
Abhilash Raj -
Barry Warsaw
-
galuszkak
-
Jack Underwood
-
Kẏra
-
Kẏra -
Mark Sapiro
-
Richard Wackerbarth