[Mailman-Developers] [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3

[Barry Warsaw]

On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:

>On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>
>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>
>> Need a script to upgrade from MM2 to MM3
>> https://bugs.launchpad.net/bugs/965532
>
>Here are some thoughts on a possible migration technique.
>I would request discussion and suggestions.
>
>In particular, what about the idea of converting the configuration file to
>HTML as an intermediate file format?  Selectable css could easily render it
>as a viewable report. It could still be edited by hand without too much
>difficulty.

It's an interesting idea.  As you observed, mm2 can export to XML, so it's not
such a big stretch.

w>Steps to migrate from MM2 to MM3
>
>1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>include testing to assure that things are ready to create new lists.

Right, and it should be doable even while mm2 is still functional.

>2) Translate list configurations
>
>  a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
>  to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
>  would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
>  populate configurations at the List level. It might also reformat selected
>  parameters. In particular, various <option type="radio" > entries might be
>  transformed into enumerations such as "Yes"/"No" or
>  "Hidden"/"Private"/"Public" rather than numerical values. This would
>  enhance readability.
>
>  b) TOOL3 would populate the World level with the MM2 defaults and
>  recursively promote common values up the tree, leaving only those entries
>  which would need to override their parent to derive the current
>  value. Values which match the parent would be flagged. (The inheritance
>  flag should be tri-state. "Differs from parent", "Same as parent",
>  "Inherited from parent")
>
>  c) At this point, the user might edit some of the configurations and rerun
>  TOOL3 adjusting the inheritance flag as appropriate.

I think the trickiest part will be what to do about subscriber information.
In mm2, this is always list-centric, but in mm3, you need to collate and
globalize all the membership information into the user database.  You can
probably do the same kind of up-promotion there, but it would be from
member->address->user.  IOW, if you see an address subscribed to a mailing
list with the same values across all those lists, put the preferences in the
user.  What happens if you see anne at example.com subscribed to three different
lists with three different passwords?  That's a tough one because there's no
way to express that in mm3 (nor probably should there be).

So I think you will occasionally have to just resolve some conflicts by
flipping a coin.  In the case of passwords, perhaps you'd always make the user
do a password reset.

>  d) Now, we begin translation to MM3 configuration options. For each MM3
>  option, TOOL4 computes the equivalent value from the MM2 values. Each
>  computed value also gets the corresponding inheritance flag. Values that
>  cannot be computed from the available information get the "Inherited from
>  parent" flag. MM2 values used in computations are marked as "translated".

The user herself could probably write a script for this pipeline you're
proposing, that would allow her to do bulk transformations of configuration
variable.

>  e) After a chance to edit the MM3 configurations, TOOL5 would recompute
>  inheritance flags, report any MM2 values that have not been translated and
>  produce a copy of the configuration file simplified by removing all
>  inherited entries.
>
>  f) After a final inspection TOOL7 would actually import the configurations,
>  committing entries to the MM3 database.
>
>3) For the migration of rosters, we should be able to do it one subscription
>at a time through a pipeline that permits pre- and post- hooks.  A --dry-run
>option would be appropriate.

Almost definitely.  The --dry-run step which would produce an output of those
conflicts, and impossible situations.  The user would then have a chance to
re-edit the intermediate file so that the values can be better mapped to mm3.
It's probably worth doing for both the rosters and list configurations.

>  a) We can assume that each email address is a distinct person.  The
>  subscribers can utilize the UI to merge email addresses into a common
>  persona.

We'll probably need a "claim and merge" operation in the system.  And a way to
purge unclaimed addresses after a while.

>  b) We can also assume that each subscription overrides its parent in the
>  Persona-->EMailAddress-->Subscription hierarchy. The individual users can
>  use the UI to consolidate their selections.

A good challenge for the Postoriusians :)

> All of the tools should be written in Python, hopefully in a dialect that is
> common to all of the versions supported by MM3.

+1.  Today that would be 2.6 and 2.7.

> TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes
> as they come in. Similarly, in step 3, we can use TOOL1 and discard the list
> configuration information.

> TOOL2 can reformat the XML as HTML, thus making the input data into a
> viewable report. The inheritance flag would become a class attribute on the
> option. Would it make sense to go a step further and generate html forms and
> run a trivial http server on localhost? It might be easier to do this in
> django, but I think that requiring that level of installation is probably
> too much for the current situation.

It wouldn't be hard to do in standard Python.  OTOH, I'm not sure we want to
maintain a stack of html templates, forms, and form processing in the
conversion tool.

It's sounding like this suite of conversion tools may need to be a separate
sub-project.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20120410/d30948b6/attachment.pgp>